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The Future of Text Vol 4, 2023 


All articles are © Copyright 2023 of their respective authors. This collected work is © 
Copyright 2023 Future Text Publishing & Frode Alexander Hegland. 


A PDF is made available at no cost and the printed book is available from ‘Future Text. 
Publishing’ (futuretextpublishing.com) a trading name of ‘The Augmented Text Company 
LTD, UK. This work is freely available digitally, permitting any users to read, download, 
copy, distribute, print, search, or link to the full texts of these articles, crawl them for 
indexing, pass them as data to software, or use them for any other lawful purpose, without 
financial, legal, or technical barriers other than those inseparable from gaining access to the 
internet itself. The only constraint on reproduction and distribution, and the only role for 
copyright in this domain, should be to give authors control over the integrity of their work 


and the right to be properly acknowledged and cited. 
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Reading this book in ‘Reader’ 


If you are reading this book in our macOS software ‘Reader’, which was developed as part of 
the Future of Text Initiative, you have more ways to interact with it. We hope to have similar 


capabilities available in XR soon, and then to go far beyond. 


The Future of Text [1] The Future of Text [2] The Future of Text [3] 


Click on the number in hard brackets and then the title in the pop-up to load each volume 


a future of text, AI generated! 
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Full Screen & Outline 
¢ [ESC] to quickly toggle in and out of full screen 


¢ 3-—(minus) to fold the book into an outline 


Finding 
¢ 3-F on selected text will change the view of the document to show only lines with that 


text, plus headings 


Interactions, including with AI 


* Control-Click for a small menu of useful options 


More Useful Arrow Keys 


Right and left arrow keys work as normal for next and previous page, but up and down arrow 


keys work for articles: | down arrow for Next Article and up arrow ¢ for Previous Article 


Advanced Views (ESC to Exit) 

* d-shift-K to see headings with Keywords in colour 

¢ d6-shift-N to see only Names with headings for context 

¢ d-shift-D to see any Defined terms (terms in this document’s ‘Glossary’ ) 


¢ Select text and [spacebar] to see the text full-screen, single-column for easier reading 


¢ For more on how to use Reader, including how to Copy as Citation, click on the heading 


‘Reader Appendix’ to jump to that appendix and you can do 3-[ to return to this page 


Get Reader for free: http://www.augmentedtext.info 
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The stakes are simple: XR extends our potential. 


How we choose to extend ourselves defines who we are. And who we want to be. 
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The Future of Text 


The Grouc’. ~ §yf@nline | 2023 


The Future of Text Symposium 2023 
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Welcome to Volume IV 


Our intention with everything we do is to promote the thinking that, even though text has 
been critical to expanding human thought for millennia, with digital text we can go much 
further than we have done so far, with AI further still and with XR/VR/AR even further. If we 
do not augment how we interact with our knowledge through text, we risk not only being 


overwhelmed but also misinformed, particularly with the threat posed by AI. 


“The world of reality has its limits; the world of imagination is boundless.” 


Jean-Jacques Rousseau 


We are thankfully mostly in the realm of imagination of what XR can be. Let us hope we can 
imagine an augmented future together, and not just have one sold to us. Thank you to all 
those who participated in our weekly Future Text Lab meetings, the annual Symposium and 
who contributed articles to this volume. It is humbling to be among you. If you download the 
other issues from https://futuretextpublishing.com and if you use Reader (see previous 
pages), you can interact with this document in ways you cannot with a regular PDF viewer 
because this document features a Visual-Meta appendix: https://visual-meta.info. Please keep 
in mind that this level of metadata can also make rooms in VR environments possible, 


something we hope to be able to experiment with. 


Editor: Frode Hegland. Copyeditor: Jonathan Finn. Editorial Board: Alan Laidlaw, 
Brandel Zachernuk, Dene Grigar, Fabien Benetou, Ismail Serageldin, Leon Van Kammen, 
Mark Anderson, Peter Wasilko, Vint Cerf. Contributors: Adam Wern « Alan Laidlaw 
Alexander Burdiyan « Alexander Mehler « Bob Stein ¢ Christopher Gutteridge * Daveed 
Benjamin « David G Lebow ¢ David Johnson ¢ Eric Vicenti * Fabien Benetou * Frode 
Hegland ¢ Gabriel Herrero-Beaumont * Giuseppe Abrami * Horacio Herrera ¢ Ismail 
Serageldin « James Addison * James O'Sullivan ¢ Jasper Hustedt * John Bowditch * Jonathan 
Finn ¢ Joseph Waliya « Julian Gagel « Kevin Bonisch « Leon Van Kammen ¢ Livia Polanyi « 
Luc P. Beaudin * Mark Anderson * Mark Bernstein « Mez Breeze * Matthias Miiller-Prove « 
Matthew Neary Michael Pidd « Nayanatara Prakash « Patrick Lichty « Peter J. Wasilko « 
Philipp Stephan « Phil Gooch ¢ Rachel Horst * Rob Swigart * Rollo Carpenter * Sam Winston 
¢ Shaduan Babbili * Sophie Whittle * Theresa Berg ¢ Timo Liittig * Vint Cerf * Yannick 


Heinrich 
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This is a call to arms to think about thought in multiple dimensions of freedom 
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Trajan s Column 
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Opposite is Trajan’s Column in Rome, completed in AD 113, one thousand nine hundred and 
ten years ago. Notice the typeface. Quite literally written in stone, we can see how standards 
endure. Very few standards exist for how we interact with information in XR environments. 
What we design, implement and disseminate now will impact the potential for work in 
XR for generations. Think also of the personal computer which has become ossified into 
applications for writing, messaging, spreadsheets, web access and a few other specialist 
applications and games. The development of digital interactions in the 1960s has become just 
a few ways of working. With XR it is not only pixels on a rectangle—our computer screens— 
but everything in sight that’s at stake. Thankfully we are still in what might be thought of as 
the ‘Cambrian’ explosion of XR where we can still—for a short while—experiment widely. 


Questions still open as per publishing this Volume 4 of The Future of Text include: 


¢ How will we define ‘tool’? Where will the equivalent of a context menu be in VR? What 
tools will work where—will you be able to take tools with you? What about physical 
tools? What might symbol manipulation at scale be? Will we move information or 
ourselves? 

¢ What will work be? What will our bodies be? 

¢ What will links be, what about connections? 

¢ What is a space, how do we move between spaces? What is a ‘wall’, is it defined as a 
rectangle or with semantic meaning? What is it positioned in relation to; my head, my 
desk, the room? What software owns it? What can be seen, interacted with and ‘taken’ in 


shared spaces? 


23 


Foreword 


Ismail Serageldin 


This is the fourth volume in the series that this talented group of individuals has produced. 
As one would expect, the contributions range far and wide, yet they have a recurrent theme 
that seems to have enthralled many of the participant authors: namely the radically different 
possibilities that Virtual Reality (VR)/Augmented Reality (AR)/Mixed Reality (XR) — 
henceforth referred to as XR — can hold for how we interact with Text. For in the end, Text 
remains the most powerful code that humans have invented to communicate with each other 


across space and time. 


Text allows you to discuss the most abstract concepts, even those that, by definition, 
transcend the experience of humans: the divine, the infinite, the high orders of the 
multidimensional, to name but a few. The infinite capabilities of text — a manipulation of the 
limited number of the alpha-numeric symbols — had remained confined to two-dimensional 
space. What would adding a third dimension open up as new possibilities of expression, as 
new possibilities of understanding? 

For me, this is also happening at a unique moment in time, an inflection point in the 
long progression of humanity from the oldest times of cave dwellers painting on the walls to 
the amazing possibilities that are opening up today. Our ever faster and more powerful 
computers, ever better imagery will blur the distinctions between the real and the imaginary, 
to the point where we will no longer be able to trust our senses of sight and hearing to 
distinguish between the two. On top of all that, Artificial intelligence (AI) is advancing at a 
phenomenal speed, especially in generative AI, where Large-Language Models (LLMs) have 
shown an amazing capacity to emulate our manipulation of words, symbols, and images. 
Drawing on our own vast production, they can create new combinations of text, as well as 


images, that surprise, enthrall, and baffle us. 


What is more, there are several currently separate threads that can come together to 
profoundly change how we interact with text, or even think about text, which here involves 
both images and the manipulation of alpha-numeric symbols into new, coherent segments and 


wholes. These are: 


First: The capacity of generative AI, especially manifested in LLMs that have clearly 
passed the Turing test, to seduce us into almost believing that we are interacting with a self- 
aware intelligent consciousness. All that will only become more powerful and capable, as 
such models now routinely produce poems and text, and can even mimic the styles of poets, 


writers and painters. 
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Second: The voice-to-text and text-to-voice transformations that are now almost 
routine and well-nigh perfect. As are the incredible capabilities of instantaneous translation 
from one language to another. 

Third: The huge advances in Brain-Machine-Interface (BMI), where machines are now 
reading the firing of neurons in the brain, and correctly interpreting them. Paraplegic patients 
can now think of entire words — not just individual letters — that the machine can “read” and 
interpret with over 98% accuracy and at a speed that would approximate 80 characters per 
minute of typing. 

Furthermore, there are those who are working on how the brain recognizes images, and 
retains it in its memory, and developing programs that can recall these images by reading 
brain scans. 

Fourth: the enormous increase in “wearables” where we not only have our individual 
machines in our pockets or purse, but simply worn on our clothes or body (e.g. smart glasses, 
smart watches). 

As we approach the end of 2023, it will not be too far in the future when we can 
imagine thinking about ideas, which invariably must be manifested in our mind as either 
images or words or combinations of the two, and have our AI program “read” our thoughts 
and “respond” directly into our brain. This intensely private dialogue, between person and 
machine, with no external manifestations, poses profound questions about autonomy, agency, 
privacy, and the meaning of communications. And while this may still be further in the 
future than the use of XR to bring an extra dimension to our dealing with “text” in programs 
such as “author” and “reader”, they are another manifestation that we are indeed approaching 
an inflection point in our relationship with language and in our ability to manipulate text, as 
much as we are approaching a new level of interaction with machines. 

It is an exciting time to be involved in these fast advancing and far-reaching 
transformations. I am sure that many of the brilliant persons in our group have already 
thought out many of the technical questions, including the ethical and safety measures that 
should accompany the development and deployment of new technologies. Many are worried 
about how we can deal ex-ante with almost perfect fakes that can now be routinely produced 
by AI even as we are being reassured that we will be able forensically to identify ex-post the 
fake from the real. But ex-post is after the harm has been done, it does not replace ex-ante 
filtration. A perfect fake video released two days before an election can go viral and destroy 
the electoral process before trusted authorities can prove it is a fake. To me, these kind of 
dangers are far more real than the fears that the AI will somehow “take over” the civilian and 
defense systems, and like “Skynet” in the “Terminator” movies subjugate or destroy 


humanity. 
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But this is a somewhat different set of questions than those dealing directly with text 
and its interaction with humans, which remains at the heart of the present volume. For in the 
end, the production of knowledge and its proper mastery will depend on text. And how we 
interact with text, whether in two-dimensional or three-dimensional space, is the essential 
underpinning of how knowledge is created, stored, accessed, analyzed and added to. 

In the final analysis, the text (written or spoken) is the basis of all human transactions 
with other humans, and to the extent that code is a form of text, it is also the basis of all 


human transactions with machines. 


Our laws are written in text. Our most valuable cultural legacies are enshrined in text. 
The history of humanity is inscribed in text. Moreover, today, every second of every day, we 
are adding to the collective production of humanity massive amounts of text through the 
internet and the massive storage and retrieval facilities that modern computers make 
possible. Will we be better able to handle the new level, if not the new type, of interaction 
through the three-dimensional realm of XR than what we are able to do through the more 
conventional two dimensional world of paper or screen? 

And how would the new XR-mediated technology impact the already fraught situation 
of massive inequality in the world? Much has been said about the “digital divide”. Would 
this new XR-mediated technology increase this digital divide? Would the costs of the 
equipment needed, plus the complexities of the handling of text in 3-D, cause the less 
fortunate of the human family to be left further behind? Or will what we develop be more like 
mobile phones that have penetrated all but the most destitute communities, and become a 
useful all-purpose tool from Mobile banking as in M-Pesa in Kenya to tele-medicine in many 
deprived communities? 

Only through the manipulation of text can we dominate memory and dissect the 
present, foretell the future or dream of “what could be” to inspire us to go beyond accepting 
“what is” with all its imperfections. Societal self-awareness is developed by those who 
articulate text to help resolve our inner tensions and compulsions, be they poets or politicians, 
artists or scientists. 

Thus what we do as we anticipate, discuss and help advance the future of text, is 
essential for all disciplines... from those who manage text for news, to the scholars who sift 
through the massive amount of data and information to find pattern and meaning. In that 
process they make their own contribution to that cumulative output that is the basis of our 
contemporary understanding of self and society. Thus, our efforts should help all disciplines 
search, find, retrieve, manipulate and add to the knowledge imbedded in text produced by 
others, as they make their own contributions more accessible to others. And in the collective 


human enterprise as our societies face the challenges of a turbulent word, we need the 
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contributions of everyone. We need the knowledge of the natural sciences, the insights of the 
social sciences and the wisdom of the humanities. 

Finally, as we continue to advance the ability for the next generation to better interact 
with text we should continue to be bold in developing and adopting new approaches and 
technologies. But let us also remember the profound questions that T.S. Eliot asked almost a 


century ago... 


Where is the Life we have lost in living? 
Where is the wisdom we have lost in knowledge? 
Where is the knowledge we have lost in information? 
-- T. S. Eliot, the Rock (1934) 
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We can. And we must 


Frode Hegland 


At the time of writing, late 2023, the world is in turmoil. At the same time we are seeing 
new heights of technology and new depths of scientific insights. Just this week my 6 year 
old son asked me about the origins of everything, and we started talking about the Big Bang 
and he memorised the age of the universe being 13.8 billion years. We then sit down in the 
living room to look at Brian Cox and others talk about the universe, and go through some 
YouTubes where we are reminded of the recent controversial discoveries made by the James 
Web Space Telescope (JWS)? which shows that the universe might a quite a lot older, perhaps 
even twice as old as we thought>. Or perhaps not*, but we are at a point in our evolution 
where we can see billions of years into the past and have these discussions to understand our 
universe ever more fundamentally. Our sun is half way through its 9 billion year lifespan—on 
galactic scales human lives are almost immeasurably short—but human scales is all we have. 
We can look at the past and learn but we must aim for the future, and if we don’t mess it up, 
there is a long future for our descendants. 

Our species’ evolution has seen many changes whereby we ran as rodents around the 
feet of dinosaurs, climbed in trees as our common ancestors to chimps, started shaping tools 
—and our environment—and stood up on the savannah, built dwellings in villages, cities and 
states, recorded our thoughts in clay, on skin, on paper and on glowing digital screens, 
changing who we are—our capabilities to act in the world and to understand the world—at 
every step. 

We have been connected to digital information over the last half century, 
tentatively, step by step, taking much of our analog world with us, as Mark Anderson writes 
in The Unintended Long Shadow of Print in The Twenty-First Century and Chris 
Gutteridge reminds us that we are Still Working in a Virtual Straightjacket. We first looked 
at the digital world on green CRT monitors and later colour monitors, which greatly increased 
in resolution quality and size—growing both larger and smaller. Some choose to work with 
large desktop monitors while many work on smaller laptop monitors, at least part of the time, 
but also access their information on smartphones and digital watches. 

Still, these are merely windows we are looking through, we are not yet fully 
immersed in the digital dimension of our lives. 

Over the next few years we will be moving into the digital world fully. First with the 
headsets available today and coming online over the next year, which will deliver a whole 


new level of experience. We will then see Moore’s Law? over time improve display and 
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sound quality, haptics will be added and weight and cost will go down. This, I believe, will 
be inevitable. What will not be inevitable is how we will live and work in these 
environments and who they will be designed and refined to serve. Will it primarily be the 
developers of the system, as we have seen in the era of windowed computing, with Microsoft, 
Apple and Google? Or will we be able to organise so that the actual user will have a real 
stake in the development? Fabien Benetou argues Why closed products aren't enough for 
a creative & liberating future of text and I agree. 


As for the future of text, all I can tell you is that the future of text is to continually 
question what the future of text can and should be, as my professor Ed Leahy from Syracuse 
University reminded me so many times. His time is past, as is the time of my mentor Doug 
Engelbart who informs so much of my personal vision of what the future of text is. But your 
time is not. 

We are On the Eve of the VR Paradigm Shift and it is not enough to close our eyes 
and imagine what might be presented. It is also not enough to wait for news from Apple, 
Meta, Microsoft or Google as to what they believe they should sell us. We need to think 
anew, and we need to read the wisdom of those who have thought about text for longer than 
the latest systems have been available, we need to pay attention to the perspective of pioneers 
like Mark Bernstein, and the originality of people like Alan Laidlaw as well as look at how 
our extended reality will be informed and shaped by the concurrent development of AI, as 
outlined by Daveed Benjamin. There will opportunities far beyond ‘floating screens’ and 
we can—and should—think about our infrastructures anew, particularly for how we can 
address information, in both senses of the word (point to information and talk about 
information) and how we can evolve new perspectives as Peter Wasilko describes. 

As we step into this new dimension of display, interaction, experience and thought, we 
can shape the world and we must do this together. 

The cost of leaving this to any other party is great. If we do not take charge as 
‘knowledge workers’—an admittedly very broad brush, and leave it only to those selling 
systems, we will severely constrain our choice and the opportunities for thinking different®. 

I thank all the contributors for helping move this dialogue forward and I thank you for 
reading this, no matter what time period you are reading this, whether it is when we first 
publish or long after headsets become a normal part of the work day, or perhaps long after, 
when we are fully immersed through other means. Maybe you will find something to inspire 
you to build better thinking environments to truly augment, rather than just passively 


entertain. I hope so. 
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We have not had an opportunity to design how our brains function. 
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We now have a chance to design how our brains connect. 


XR & traditional digital environments with AI offer tremendous potential. 


We should take this very seriously. 
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© PART 1° 


(pre-Apple Vision Announcement) 


This section of the Future of Text is the last authored before Apple releases their expected 
VR/AR/XR headset which we think will be called ‘Reality Pro’ and was released as a Journal 
issue before being included in Volume 4. These are therefore the last words we have been 
able to write down unfiltered by what Apple’s implementation will be like. The final rumours 


are these: 


“The Worldwide Developers Conference (WWDC), Apple's annual developer and software- 
oriented conference, is now just a few days away. As with previous years, Apple is expected to 
introduce major updates to all of its operating systems, but unique to WWDC 2023, Apple is 
also very likely to unveil xrOS, the software that will run its much-rumored AR/VR mixed- 
reality headset. Here's everything we're expecting from it. Apple has not publicly 
acknowledged the existence of its headset, but internal Apple sources have confirmed that the 
operating system will be called xrOS. xrOS is believed to have an iOS-like interface that will 
be familiar to iPhone and iPad users, but presented to them and interacted with in an entirely 
new way. We are expecting a dedicated Home Screen containing apps and customizable 
widgets, all of which can be rearranged, and all viewed on dual high-resolution 4K micro 
OLED displays with up to 3,000 pixels per inch for a rich, realistic, and immersive viewing 
experience. Both of the user's hands and eyes will be tracked by more than a dozen optical 
cameras, and the wearer is expected to be able to look at an on-screen item to select it, and 
use a hand gesture like a pinch to activate the item on the screen. The xrOS software could 
even potentially project AR app interface elements onto actual objects in the real world for a 
mixed-reality overlay kind of effect.” 


macRumors website (archived as a PDF’) 


The step into working in fully immersive technology has started long ago, but I believe that 
Apple has a corporate ‘knack’ of delivering powerful technologies which become welded to 
the user, to a degree nobody else has. Witness the iPhone and the Apple Watch, while not 
ignoring the Mac that started it all for Apple. Meta won’t be able to do this because they 
don’t have access to the user’s work data in the same way Apple has, so this won’t be a 
‘metaverse’. Microsoft can’t do this because, well I honestly don’t know why, but let’s see 


what happens on the 5th of June 2023, a week from today, when Apple releases their headset. 
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A technology which fully envelops the user has a much higher degree of ownership of 
the user and this matters. The more we are locked in, to companies or technical 
infrastructures (from file formats and network protocols to ways to render scenes), the less 
freedom there is for the ecosystem to evolve. This step is too important for anyone to ‘own’ 
or dominate. 

At Syracuse University a professor, whose name I do not remember and whom I have 
not been able to track down, said: “Truth Kills Creativity”. 

And this highlights how difficult it is to dream. Once the headset is announced, we will 
collectively think about some interfaces and infrastructures that this is ‘just the way it is’. Let 
us never settle for ‘this is just the way it is’. Let us always strive for better. 

The articles contained herein are intended to help you, in the future, get some unbiased 
perspectives on what might have been—what might be. 

This matters. VR (including all flavours, from completely immersive to only slightly 
overlaid on our physical reality) will be where we will spend much of our work time, so how 


it extends our minds and how it limits our minds will matter. 
As I write in my piece on VR & Al, I believe that VR can truly help us harness the 
power of AI, instead of letting us simply be overwhelmed. 


Soon we will be escaping flatland. 


A Powerfully Open Future. Dreaming is important, and so is building an evolutionary 
environment where dreaming matters, where dreams can become real and not remain mere 
fantasies. This is why the most powerful thing I wish for, for the coming VR, is that we all 
work to make it as open as possible, to enable multiple imaginations to compete. 

The Internet was not a new network, it was a standard for networks to connect, hence 
the name “‘inter’-‘net’. We will need an internet for VR the same for the virtual worlds now 
being unleashed. We need multiple infrastructures for those of us passionately working to 
unleash the augmentation potential of VR to be able to experiment to experience new ways of 
working, thinking and communicating, allowing different approaches to truly compete. 


You are the one actually dealing with this, we were only dreaming. 


Good luck to you! 
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Still Working in a Virtual Straightjacket 


Christopher Gutteridge 


The modern stories around the use of VR seem to be: 
* gaming 
* museums and exhibitions 


¢ shared meeting and collaboration 


My limited experience of a virtual meeting space was that it entirely missed the power of VR. 
VR as a productivity tool for managing data and documents is generally like simulating all 
the aspects of the real world that make a digital library and desktop better. We add back in the 
walls obscuring things, and the time and effort to navigate. 

In a shared space (or a gaming world) it’s useful to make entities behave in ways that 
other observers can understand, but in a personal VR space this isn't needed. It’s trivial to 
zoom nearly infinitely, it's possible to move at any speed and rotate the world to any angle. 
This isn't user friendly but neither is Blender or vi. I don't want to spend 3 minutes walking 
around something. My time is valuable to me. It's why I learn the keyboard shortcuts. I gain 
nothing from looking at a virtual art gallery that I couldn't get from looking at photographs... 
or at least at a 3D model spinning in a webpage. 

The issue with VR is that it's still the answer to questions people are not asking. It's 
possible it's never going to be the answer to a question, but more likely that we don't ask the 
right questions, or at least not loudly enough. Often to find the “real” application for 
something, we first have the “killer app’ —something that is actually useful and makes it 
worth people engaging with. With Linux it was the GIMP art package and Open Office. For 
the first time you could do common IT tasks on a purely free software stack and that made it 
worth people installing on the desktop. Frankly, VR is yet to have a killer app. I've never 
once had a problem and thought “this would be easier if I had a VR rig”. And until it does, 
we're not going to get the unexpected new uses. It’s a popular story that SMS was initially 
considered something people would only occasionally use, and that is a formative tech in 
modern youth culture. The 303 synth was designed as a bass guitar replacement <https:// 
www.soundonsound.com/people/history-roland-part-2> which is was crap at, but, again, was 
used in ways the creators never expected when placed in the right (or wrong!) hands. 

Akey difference though is that the barrier to finding novel uses for SMS and the 303 was 


very low. VR has a lot more issues... it’s been very expensive, it’s kinda dorky, and it makes 
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the wearer genuinely vulnerable so is only suitable for use somewhere the user feels secure. 

It had a unique chance during the pandemic, when people spent more on 
communication and home entertainment hardware, but it was technology like Skype which 
had the massive boost. People wanted to still have some face to face time with coworkers and 
friends. Although after six months my team opted for mostly camera-off meetings for internal 
stuff. It wasn’t uncommon for someone to be wearing headphones listening to a report while 
putting out the laundry or similar zero-brain chores that the body can be doing while the mind 
is doing the day job. VR is the antithesis of this, it ties you to a location and demands ALL 
your focus. 

It has obviously found some high-end uses like telecontrol of equipment, but that’s not 
the killer app that lets 15 year olds innovate with it. 
Lots of world builders exist—Second Life and its imitators, but these feel dangerously close 
to NFT grifts. Selling virtual land to foolish speculators. But what doesn’t exist (to my 
knowledge) is tools which let non-experts innovate with the interface itself and the rules of 
the world. Kids should easily be able to define the physics and interactions of a virtual world, 
and share them with each other to remix and build on. That’s when we’ll start to see the 
things we never anticipated. Until then we’ll just see more middle aged people like me trying 


to recreate our offline experiences online. 
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Text with Graphics 


David Johnson 


VR or AR will allow the combination of text with graphics. The result will be the use of 
inference arrows. Hierarchically nested. VR will allow an infinitely zoomable canvas. And 
the arrows will be computable. And shared spaces will enable debate in the form of iterative 
construction of inference arrow structures. We will see the emergent complexity of our 


collective thought. 
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Goodbye to My Old School Large Format Display 


David G Lebow 


The advent of VR systems capable of providing the equivalent of a wall screen display with 
interactive features for moving text and images through gesture promises to transform how 
people engage in multisource sensemaking. For anyone who is excited to exercise their 
curiosity by tapping into the internet but feels paralyzed by information overload and 
information anxiety, this is great news. 

Many years ago (i.e., pre-digital age), I remember reading an article with some 
skepticism about how people someday will carry the equivalent of the library of congress in 
their pockets—around 15 million volumes weighing many tons. Today, anyone with a 
connection to the internet and a Webphone in their pocket can access almost everything ever 
published on almost any topic—the storehouse of human knowledge at your fingertips. 
Paradoxically, as the amount of information available to us through digital devices rapidly 
expands, more and more people are accessing this information on tiny screens. 

Back in 1992, as a graduate student in educational systems and technology, I began 
work on my first paper for publication. After reading over a hundred documents in hard copy 
on my topic of interest, I felt overwhelmed by the volume of information. Out of 
desperation, I copied all of the fragments of text that I had highlighted in my 100+ source 
documents to a Word file using my Mac LC with its 12” screen. I printed out the results, 
which included 88 pages of text fragments with document code numbers and page numbers 
for reviewing fragments in their sources. I cut the fragments into snippets, put the snippets in 
a shopping bag, shook the bag to mix everything up, and then used the floor of my office as 
an old school large format display. Sitting on the floor, I reached into the bag, pulled out a 
snippet, and placed it on the floor next to related snippets. As the process unfolded, I 
constructed a narrative-like structure by organizing snippets into columns of related 
fragments and arranging the columns into rows that reflected an emerging storyline. 

After three iterations of organizing snippets, reviewing the results, filling in missing 
information, and looking for non-obvious associations and patterns, I had an Aha moment. I 
saw something that was a new observation. 

Arguably, advances in VR technology combined with large language models and related 
technology promises to deliver on the promise to democratize the creation of new knowledge, 


as first imagined by internet pioneers. Goodbye floor. 
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Virtual Reality 


Deena Larsen 


“Imaginary gardens with real toads in them...”” Marianne Moore. 


I, too, mistrust it. There are things that are important within our own 
reality. 
Experiencing it, however, from our own imperfect understanding for it, we discover 
that there is in 
it after all, a place for the meaning. 
Hands that can grasp, eyes 
that can see, feet that can move 


and adjust other realities, these dimensions are important not because a 


fascinating new technology can put a glamour on them but because 
they are 
useful; when they become so imaginative as to become 
insightful, the 
experiences may provide visions for all of us—that we 
would not otherwise have come to 
or could not understand. The gloves, 


grasping on in thin air or in quest of something to 


hold, golf clubs swinging, an adventurer storming a moat, a reader harvesting 
these words, 
the program trying new algorithms like muted conversions in dimly lit 
bars, the code— 
(s)talker, the generator—case after case 
could be quoted if 
we build it; nor is it dated 


to cite use-cases for military or education or 
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gaming; all these phenomena are important. One must make 
a judgment 
however: when dragged into commercialization without context, the result 
is not virtual reality, 
nor till the programmers among us can be 
“purveyors of 
real meaningful content” —above 
likes and views and selling and can present 
for inspection, virtual hallways with real people in them, shall we 
have 
it. In the meantime, if you demand on the one hand, in defiance 
of their prophecies— 
the imagination of life in 
all its potential, and 
that which is on the other hand 


vicarious, then you are immersed in virtual reality. 


41 


Why closed products aren‘t enough for a creative 


& liberating future of text 


Fabien Bentou 


Yet another release around the corner. BigTech is once again promising utopia or at least a 
better, faster, more efficient and creative at the same time future. Chances are it will be a 
partial fantasy and a partial truth. The product will definitely be mindfully crafted by experts 
in their respective field with access to resources most of us barely dare to dream of. Yet, 
behind the promise hides few things we must always be cautious of. 

The first and most fundamental challenge is daring to dream beyond what is offered. 
The product might be slick, might have perfectly finished edges, still it is not finished. One 
could imagine all participants of the project and they would surely admit that they had to deal 
with constraints, at the minimum physics, delivery date and budget, but one can imagine a lot 
more. Consequently whatever is being sold is simply an imperfect version trying to solve 
these constraints. At the moment of its release it is already outdated from a technical 
standpoint. One should thus look at the result of the result as a stepping stone to do better and 
question, not merely accept, what it can actually do and how one could do, with or without it, 
better. 


Secondly and sadly a recurrent challenge is considering the true cost. There is of course 
price as the immediate cost. It is what will already prevent most of us from buying from 
BigTech high-end product that are playing on an elitist appeal. We want to look cool, all of us 
no matter our age or background. We all need deeply acceptance from our peers but even 
from complete strangers. Having, and showing, products that represent the state of the art in 
tech does, from most, just that. They make us look like we are already present in the future. 
At least that is the value proposal sold by most advertisements for BigTech. Again this is not 
a complete lie, it does indeed provide radically new usages and thus can help us do more, or 
better. Yet what always comes attached to the price are locks. We know of such explicit locks, 
where for example it is not possible to install an application not part of the store owned by 
the company that made the device. We also know of DRM, or digital rights management, that 
makes our own devices prevent themselves acting as we may want, think how you cannot 
download a book to play on an ebook reader without the right software. What is more subtle 
though and what we might miss is how these locks interplay. What if instead you could 
indeed bring your ebook or music to another device with ease? Well of course the trick comes 


from the company that made the first device itself, and is obvious: buy a second device! Both 
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devices will perfectly connect together, providing you a smooth experience, something 
almost ethereal where data have no place, where one can “think” across devices. The 
“solution” is apparent and yet we tend to forget about that cost, namely lock-in. The better a 
device can simultaneously work with “authorized” devices from the same seller while 
preventing use from any device, the deeper lock-in gets. The more helplessness we feel as we 


know our dependency on the device and its ecosystem grows. 


Thirdly and finally the most oppressive challenge comes from ourselves, after years of 
consumption, namely learned helplessness. Year after year, using more and more amazing 
products that seemed to complete and that deepen lock-in, we learned to use rather than 
create. Sure we did learn to create within the boundaries of the products we purchased, 
carefully trying to think away from, yet still within a box. We assumed that BigTech products 
are complex, which they are, that they are efficient, which they are, that they are covered by 
patents, also correct, and that making our own would be prohibitively expensive and even 
technically impossible. This though is all partially true and partially wrong. On our own, it is 
indeed nearly inconceivable to watch a BigTech release announcement and think “Ah, 
amazing, I could have built that!”. This is actually true not just because we do not have the 
resources but because we forget how it is made and its limits. Behind the announcement one 
could imagine the rolling credits of a movie, endlessly going on and on, showcasing the 
countless experts involved. So yet, on our own, with little resources, doing a copy of what 
has just been shown with grandiose claims and much glamour is usually unreachable. Yet, 
maybe one part of it, an “easy” part relative to our own expertise is easy. What if we were to 
document ourselves on the state of the art for that part. What if we asked a friend or colleague 
about that other part, chances are for them it might be feasible. Would the final result be 
identical? Surely not but the point here being that just by doing so, we did dare to ask 
ourselves how we could do it, not if. This very step then opens up endless possibilities. The 
product showcased during the release might be amazing but is it truly what fits our needs? 
Surely not, as it was done for a theoretical user, not us with our own idiosyncrasies. 

So... yes, releases are exciting but do not forget they are ploys to make one 
intellectually salivate about the next box to put you in. What is more exciting is how WE, 
together, can make transparent documented bricks that connect. We do not need releases, we 


just need to build! 


43 


How VR can—no, will—save us from Al 


Frode Hegland 


In conversation with Barbara Tversky and Anni Rowland-Campbell this week something has 
become very clear: XR (AR/VR) can provide a powerful environment for how to deal 
with AI, particularly in education. VR environments can provide the space needed to 
unlock the potential of AI to augment how we think and communicate, whereas traditional 
computer displays and speech interactions—though useful in so many situations—severely 
constrain our interaction and therefore our involvement, and ultimately, our control of AI. 
The following is a review of our possible near-futures and a suggestion and plea that we take 
it seriously and invest in augmenting how we think and communicate. 

From a perspective of valuing ‘Intellect Augmentation’ (IA), in the Doug Engelbart 
tradition, as any new and powerful thinking and communications technology, from paper to 
print to the Internet, AI can both increase the depth of our thinking and also make it 
shallower. It will all come down to how and when students, academics, and people in general 
use it. Engelbart presented the notion of augmenting human intellect in his seminal 1962 


paper, which would be the basis for his ‘mother of all demos’ in 1968: 


“We need to improve how we augment a group’ (small, large, internal, global etc.) 
capability to approach urgent, complex problems to gain more rapid and better 
comprehension (which can be defined as more thorough and more critical) which result in 
speedier and better solutions (more contextual, longer lasting, cheaper, more equitable etc.). 
And furthermore, we must improve our improvement process (as individuals and groups).” 


Doug Engelbart [4] 


Anni made me consider what many people have been doing lately, thinking about AI’s impact 
on human thought, particularly as related to education. In a sense, AI can be seen as being 
mostly in competition with IA in much the same way heavily processed, fast-digesting 
sugared fast food is in competition with healthier, more whole food which the body needs to 
take time to digest; more fibre, more bran. Many of us will mostly prefer the fast food. And in 
many cases this makes sense, there is much thinking we do on a daily basis which is not 
really helpful, such as remembering phone numbers and turning on hand brakes (electric cars 
do not need them) and so on, tasks which systems have already removed from our daily 
grind. 

Historically, sugar is not all bad. After 1800, imports to Britain soared and the change 
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from a fibre-heavy to a sugar-heavy diet, noted Robert Fogel, a Nobel prize-winning 
economist, “raised the proportion of ingested energy that [could] be metabolised”®. 
Furthermore, Fogel reckoned that “bringing the ultra-poor into the labour force [and] 
raising the energy available for work by those in the labour force” accounted for about a 
third of Britain’s economic growth in the 19th and 20th centuries. Will AI augment more 
people to be more productive? For much work, it seems that AI can at least remove some of 
what slows us down, the daily grind, maybe clearing up enough mental load to give us more 
capacity for further mental exertions. 

For what we consider important information we’ll need more ‘brain-bran’. More brain 
involvement for the brain to truly understand & be involved. The breakfast cereal analogy is 
not entirely silly, the idea is making information great for digestion and integration into our 


current knowledge, as well as palatable (pleasant to interact with/providing ease of use). 


When it comes to augmenting our intellect, a lot more will be outsourced to AI in the 
future than it is now—or perhaps the term should be ‘delegated’—that much seems 
inevitable. Much of this outsourcing will be of value, similar to how we could search the Web 
for answers before, we didn’t need to go down to the local library. There will also be issues of 
actual dumbing down where people will not want to learn but get the information in a 
presentable form to solve a problem, such as that of a school paper, keeping the student’s 
knowledge shallow. We might also learn to outsource in ways inspired by the octopus’ 
tentacles which operate at a level of independence so far alien to us [5]. 

On the other side of the potential of AI, to increase our depth of knowledge, I heard a 
story of an old grandfather hearing about the recent advancements of AI and his response was 
something along the lines of “this means everyone will have access to great intelligence, it 
will almost be like we will all have a genius best friend” and that is nice. And I heard this on 


TikTok, another disruptive platform and interaction. 


linearity & non-linearity 

Human thought is multidimensional, as knowledge itself is. It can be argued that academic 
discourse is based on linear arguments however. A student cannot simply drop a knowledge 
graph or a pile of index cards on the desk and call it an academic paper. Well-written 
academic papers, at any level of education, are intended to make clear the author’s thinking, 
something which AI has become adept at mimicking. Augmenting the documents to enable 
the reader to unlock the non-linearity of the author’s thinking has been a key aspect of my 
work on Visual-Meta and now this approach looks like it can help with the problem of 


knowing if the student truly wrote a paper. 
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If the teacher, advisor or even journal editor had access to the author’s connected 
knowledge network, connected with the paper, they could use it to check the reasoning and 


connections. Not by itself, but as a connective tissue of the main work presented. 


At this point in time we need to ask what we want our educational system to do. 
Education has always been a reflection of what those with the means to control it would like 
it to be. 


teaching is what society wants taught 

How we feel about the current state of education will very much come down to how we feel 
education fits within our political views, where the Left seem to have more affinity to higher 
education and the Right being more suspicious. This is nothing new, education has always 
reflected what society wants. 

During the 1400s, 1500s and 1600s when Europe’s universities were growing, teachers 
were growing in demand. An illustrative example of this is Bologna, in Italy during the 1500s 
where there was a high demand for notaries, officials and secretaries to cater to the 
institutions of the elites as written by Andrew Pettegree in The Book in the Renaissance [6]. A 
result of this the profession of teacher increased to fulfil this need, where what was taught 
was what was needed to perform this work. In France invention of print started to challenge 
the preeminence of cathedral schools, where many municipalities appointed their own school 
masters in response to church control. Although starting with basic literacy, by the from the 
1500s these schools also provided a more liberal arts education and the institutions grew. In 
Bordeaux it was proudly proclaimed that the new ‘collége’ would “increase the profit and 
honour of the city”. Pettegree further highlights how citizens were proud of being able to 
educate the young in their own town, though this should be done in the ‘Parisian Style’, 


helping bring uniformity to the teaching profession. ‘ 


whole human thinking 


Back to today, we—as a society—have to choose to enable our educational institutions teach 
AI as a component of IA, and support how they can do this. One way is to make the linear to 
non-linear interactions more tangibly real, we need to keep cultivating the inclusion of the 
whole human in human thought, not focus on only the brain accessing information through 


small rectangular computer ‘screens’. 


Barbara helped me consider the layout of information. I introduced her to our new, 
streamlined ‘Library’ in Author and Reader and she pointed out that when she works she has 


different categories of information in tabs in a browser, in various open documents and so on, 


46 


which helps her have a sense of what’s where. In The Extended Mind [7], Annie Murphy Paul 


presents numerous examples and perspectives on this. 


This is so very important. We should not simply outsource thinking from our brains to 
AI, we should move thinking out of the exclusive domain of the perspective that it only 
happens in the brain and accept and embrace how our full our bodies and environment is the 


space of thought. 


transformed text 


Although AI already produces powerfully compelling speech and imagery, I believe there is 
still space—and need for—text. Text has been uniquely transformative for human thought for 
millennia, and can continue to do so, if we work to truly unleash text, including AI powered 
text, in richly visual environments. I do not fear a ‘textpocalypse’? but uniform columns of 
black on white text blurring into grey has probably had its day, outside of very specific uses, 
text will have to evolve to stay relevant in our work and daily lives. 

Text will have a profound impact on our future with AI and VR, if we work to make it 
so. If we do not, text will drown itself through producing too much of low value and 
producing little of use. 


But why text, in a world of powerful AI where we carry on vocal interactions with our 
AI systems and receive information in video form? Please forgive this perhaps a little 
defensive section on why text matters. Feel free to skip it if this is obvious to you already. 

Text vs. Voice and Video. A unique nature of text is its fixedness and instant sequence 


overview in a way which other media, such as voice and video cannot provide. 


The expectation of coherence in a written work is high where as the exception of a 
discussion is quite the opposite, the expectation is a live probing for understanding where the 
participants speak what is necessary to be understood, with the expectation of the listeners to 
ask questions to build a more complete mental model of what is said. A written piece has the 
expectation of being edited to some degree, or at least having been ‘looked over’. A recorded 
video is a series of images with little help for editing, which is why when editing video with 
speech, new systems which allow human speech to be re-created, uses text as the interface; 
“Tt takes only a few minutes and you'll be turning any text into natural-sounding Voice Over 


audio ”!°. 


When we prepare a spoken presentation for anything longer than what we would say in 
a back-and-forth conversation, we still refer to having a ‘script’, we still use text to note 
down our thoughts to make sure we present them in a ‘thoughtful’ order. I can testify to the 


mental effort of producing a coherent linear text, such as the one I am working on for you 
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now and my PhD (still in its Corrections phase), as being mentally taxing, requiring copious 
amounts of water, coffee and maybe some chocolate and protein bars to complete. Writing 
anything longer than a sentence is hard, but it is considerably more achievable than not doing 
so, by trying to do it all ‘in my head’. 

Text therefore offers different dimensions of knowledge traversal, able to connect in 
multiple dimensions. Emoticons will only ever achieve a different richness than what text 
provides; more amusing, more visual for a short ‘sentence’ but less parseable in longer form. 

Still images, moving images—both captured and synthesised through 3D or AI—are 
also powerful communicators in skilled hands and will become even more so over time. 
There is also nothing to replace a face-to-face discussion. I am not writing this to put down 
other aspects of human communication, though I do ask you to consider how this document 
would appear if spoken, as sound only or with images. It would be handy to listen to as a 
podcast episode but it would use a single channel to your brain, the audio stream or sequence 
of images. As text you need to make an effort to read it but you gain instant access to the 
overview of the text, your eye can jump around on the display to see connections and jump 
back to what you might now understand in a different way and more. Just as I would love to 
discuss the issues here over coffee or over Zoom, the record of such an interaction would be 
rich in human interaction but poor for access for those who would later listen in, as my 


experience in 200 sessions of the Future Text Lab can testify. 


Speech matters powerfully but it serves a different need and leaves a different record to 
what is written. 

Text vs. Images. What text uniquely brings to the table compared to other visual 
images is how it is a series of ‘grammatical symbols’ which are fixed for the user, not ever 
flowing like video or speech. Reading allows the eye to jump around and reflect on different 
parts of the text, pause at will and not have lost any context. 

Text, though visual, transcends the specifically visual. The word ‘squirrel’ captures all 
types of squirrels in all types of situations and concepts. A drawing or a photograph of one, is 
just that, one specific squirrel in one specific setting. It is often reflected upon that ‘the movie 
was not as good as the book’ and this is because the book was projected in the readers 
imagination, with just enough detail and vividness to create an immersive experience, 
whereas a movie ‘pre-rendered’ everything, never allowing for a a scene to happen without a 
previously arbitrary background not to be filled in. 

Digital Text. There was a powerful transformation when text moved from being 
primarily analog to digital. When we left working on paper for working on screens, we 


gained interaction in depth but lost interactions in space. 


When working on paper we could easily move several pieces of paper on a desk or 
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stick them on a wall, even stretching thread between them to show relationships as so many 
TV shows illustrate. The interactions digital systems afford us provide rapid access to 
resources, all within the confines of the space of a regular-size book open on our table. We 
can no longer dog-ear a book and look at it on the side to see what sections are of interest. 

Writing and image making have become separate, with ‘word processors’ and ‘drawing 
programs’ occupying separate places on our screens and using different file formats!'. Text, 
images, video and audio are not more powerful than each other, but become more powerful 
when usefully combined. 

The unique nature of digital text is in its ability to instantly change what is displayed— 
flexible fixedness. 

The cost of a digital display—though continually falling—is significantly higher than 
that of paper and is practically restricted in ways paper is not. A visual artist used to carry 
around a large portfolio case to show their work, now it’s on an iPad or on the Web, along 
with the clutter of the Web browser interface and the client’s other tabs and windows. Now a 
portfolio can provide instant access to innumerable images and also include video, but unless 
the viewer uses a large display, the image will be as small as a printed picture in a frame on 
the table. 


Text not only lost space, but also texture when it became digital. Text on paper can be 
whatever size or shape you want it to be, and it can overlap and part of other media in 
anyway you prefer. Pen on paper, in the hands of someone with the required skill, offers great 
freedom of expression. 

This all brings to mind the modes of hypertext and discussions around tools for 
thought: text for thinking and text for communicating are very different things, with an 
overlap that can get messy. Text for presentation may very well, in many or even most cases, 
be more useful if rendered into speech or video whereas the ‘authoring’ or writing still stays 
as text. Conversely, reading the text may very well be best as audio or video, though recourse 
to a written form may be better for examination of the text rather than the time-based audio 


and video where the text, in a sense, ‘washes over us’. 


dimensions of thought 


Just as there was was a powerful transformation when text moved from being primarily 
analog to digital, there can be—if we work to realise it—another powerful transformation 
when text moves from two dimensional substrates to full-space when experienced in a 


headset. 


VR can provide a rich experience for the next stage of interactive information spaces. 
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VR has none of the constraints of portable little rectangles, large and fixed rectangles or 
physical substrates. 


When the quality of the headset display becomes good enough!”, we can have the best 
of both worlds: display surfaces in the virtual world we can scale to any size, put in any 
location and connected in any way we want, at low effort and zero additional cost. Not only 
will we be able to expand on Barbara’s workspace, we will be able to transcend it, shaping 
our knowledge into truly useful forms. We have long been able to take inspiration from 
science fiction [8] [9], and now, even in mid-2023, we can start building and experimenting 
to experience. I started working with digital video when 320*240 pixels was considered 
pretty good and 640*480, which came soon after, was considered standard video. My current 
video camera is capable of 4K (4096*2160 pixels) and I’m eagerly eyeing 8K (7680*4320 
pixels) which is no longer in the realm of only major production houses. I expect the same 
will happen to VR and AR experiences. The current Meta Quest’s 1440*1600 will soon be 
overtaken by Apple’s Reality Headset’s 4K (4096*2160 pixels). This is to say nothing of the 
computing power which drives the displays. Currently Apple’s Watch is capable of 
immediate translation with stunning accuracy and speed. 

There really is no question that VR will appear very, very real, very, very soon. The 
question is simply whether we will let this opportunity to truly invent new ways of thinking 
to get the support it deserves, or whether we will revert to re-creating the physical world 
digitally. 

To be clear, I do not see AI as only a threat, a dumbing-down system, a propaganda 
machine or a way to produce what some has called more grey goo text!3, though all of those 
scenarios will likely play out to some extent. AI can also greatly help us take advantage of the 
additional dimensions VR will afford us. AI will be able to support these views, arranging 
what is shown and how it is shown with greater ease than ever before. 

To use some visual analogies AI will not be out of the picture if we make the canvas 
large enough. AI will box-in our thinking, if we frame it too small, if we interact with it 
through too small an interface. 

What will be new will be the ways the author can thread together an argument and 
present it in a way the reader can unravel and view in multiple dimensions, making it more 
intuitive to judge and experience the information. 

We need to pursue augmenting human thought with artificial intelligence in a virtual 
environment for a full brain and bodied, a fully human human experience. 

This will not only help free us from the potential of AI thinking for us, not with us, but 


it will also allow AI to augment us more powerfully. 
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To think is to move [10]. Thought is not static. Will we take this historical opportunity 
to employ AI in VR to super-charge how we think and communicate or will AI take over 
thinking and VR become mere entertainment? Will we be able to unleash deeply ancient 
aspects of what it is is to be an embodied human in a completely modern, artificial 


environment, to allow us to navigate this ‘cyberspace’ with all our faculties employed? 


‘combined arms’ 


Several years ago I wrote how we should ‘arm’ the citizenry in combat misinformation!‘, 
propaganda on one hand and increase understanding and communication on the other hand. 
Here I propose that we accept the issues we face and employ all the media available to us to 
truly augment everyone. 

The only form of control which can allow us to thrive in the face of the issues facing us, 
including both those of the world and those of the media we use to communicate with each 
other about the world, is truly distributed control. Imposed and directed control will over time 


destabilise and change actions into reactions. 


Augmenting text to augment how we think and communicate, with AI in VR, can 


profoundly lift our species. 


There really is also no question that VR will completely replace traditional digital text 
any more than digital text completely replaced analog text. Sometimes a tablet or a laptop 
will be exactly what you need and the headset will feel like an encumbrance rather than a 
liberation. Sometimes a paper book will present exactly the right reading modality. At other 
times a glance at your digital watch will be all you need—the smallest screen you own will be 
enough. To truly thrive in this new world of opportunities it will matter how we can take our 
information with us. Text in VR and text on your watch will have profoundly different 
affordances—there will be very different ways you can read and interact with the text—but it 
should retain its richness when you choose to access it through one medium or another. A 
work session in a VR room with your full library available and all your walls covered with 
relevant information, which you then synthesise into a coherent and concise ‘document’ for 


others, should be accessible anywhere for those who will read the text. 


The Future of Text 

The approach we are taking to make this possible is for a document to have what it is—who 
authored it, what it’s structure is and how it connects—spelled out in an appendix at the back 
of the document. We call this Visual-Meta and even though it can maintain all the 


information through a document lifecycle which can include version updates and even 
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conversions to other formats, and even printing to paper and scanning, it can also retain 
information in and out of VR environments and provide AI with ‘authoritative’ data through 
which to work from. Cyberspace, or the ‘metaverse’ should not exist isolated from the rest of 
our information environments, it should not exist in a ‘bubble’, but stay richly connected. My 
professor and dear friend, Ed Leahy, to whom this article is dedicated, passed away last week. 
He was that rare teacher who did not instruct and tell us what to do, he continually questioned 
and helped us think things through. Over the last few decades of focusing on the future of 
text, my mentor Doug Engelbart encouraged me to pursue looking at what he called ‘symbol 
manipulation’. I have edited and published three compendium volumes of over a million 
characters, hosted the Future of Text Symposium for over a decade, many co-hosted with the 
legend that is Vint Cerf, it is clear that the future of text will be the result of questioning every 
aspect of what it can be. Questioning what text should do for us and how text can be 


employed to most powerful effect in all its forms. 


Anything less will see text both overwhelm us and be fully displaced by more easily 


persuasive media. 


The future of text has not been written. 


Barbara Tversky Response 


We have to think of thinking as an interaction of the mind/brain, the body, and what we sense 
and create in the world. Thinking is certainly not limited to text or to words. 

Remember sensory deprivation? People with no sensory input hallucinate (they're in 
immersion tanks with eyes blindfolded and ear plugs). 

AR/VR can do some of the world part, though engineering what should be in the AR/ 
VR is far from trivial and people often don’t have insight into what would help them think, 
much less what would help others. AI might be able to help on that, depending on how the AI 
is trained and updated. AR/VR currently lacks good haptic feedback and in many cases 
doesn’t give a 1-1 distance experience. When people navigate the real world, they have a 
sense of how much time it should take, when the destination or intersection should be 
reached. 

Place, where things are in the world, is part of their meaning. We “read” graphs from 
left to right, we understand them faster and more accurately when “more” or “higher” goes 
upwards. A stable world is also important; we remember where we put things, books, 
clothing, kitchen utensils etc. are without looking and rearranging takes time to learn. For 
years I would catch myself driving to my old house at Stanford instead of my current one. 


I am puzzled by “save us from AI.” Do we need to be saved from AI? It certainly can 
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be helpful as well as harmful depending, but I don’t see how VR can save us from the 
harmful potential of AI. 


PS. 

It’s not text vs. image; if anything it’s integrating them, comics. See Scott McCloud’s 
Understanding Comics, see Larry Gonick’s wonderful cartoon guides to statistics, genetics, 
history, calculus, chemistry, physics, and more, see hundreds of graphic novels that work by 
integrating all kinds of text with all kinds of images. 

The world communicates; it’s a diagram in space (see the last chapter of my book): 
lines separating lanes for cars, buses, bikes, parking places, cross walks; arrows, pictograms 
of all sorts on signs and the roads; stop lights; the other things moving on the roads and 
sidewalks; sirens. All that guides users of roads and sidewalks rapidly and almost 
unconsciously, 

Personally, I think AR is going to be more useful in navigation and in tasks like 
assembly, cooking, knitting, preparing documents, sporting events, museums, sight-seeing 
especially for historical additions, and more before immersive VR takes off but Andreea (and 
perhaps Anni) has more experience with what VR from a chair can do. 


So much to think about and talk about! 


Frode Alexander Hegland Response 


Absolutely, fully agree. 


NOTE, at the end of 2023 J asked ChatGPT: 


How XR can improve interactions with Al and then How Al can improve XR, to get the 
AI side of the story 
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Walkthrough of a Dream 


Frode Hegland 


As I sit here in my office upstairs, at the back of the house in Wimbledon a very early 
Tuesday morning, working on my Apple Laptop, a 13” M1 Pro machine, using Apple’s 27” 
Studio Display, my iPhone 13 Pro on the desk, windows open to the bird and fresh air coming 
in through the window, sipping both coffee and water, Edgar sleeping next door, Emily 
waking on the other side of the house, I can imagine putting on my headset and continue to 
work. 

I am an unashamed Apple fanboy who read MacUser when I went to sleep when I was 
younger. I count Keith Martin, MacUser’s technical editor as a close friend, same as Bruce 
Horn, author of the original Mac Finder. I am lucky enough to have hour long discussions 
with Brandel Zackernuk every week. But I am still very concerned at how Apple will work to 
own and turn the headset into a fashion item for entertainment, sport and socialising. But 
never mind, I am writing about a dream, not a nightmare. 


I put on my headset and continue to work. 


What I was working on—the text I am writing to you now, in my Author software on 
my Studio Display—is still right there in front of me, only now it’s projected onto the view, 
perfectly aligned with the physical display. Since the room has been thoroughly scanned with 
the high resolution LiDAR Scanner in my iPhone, I can choose to not only make this virtual 
display larger, but also smaller, it can remove the physical display. 

I continue writing this document and the room I was in is much the same as in reality, 
apart from a few ambient systems giving me information about what’s going on in the world. 

I make my display a little smaller, a little taller, exactly what I want for writing, just 
using my hands and pulling on the virtual frames. I then pull out the Map of the document 
and place it on my map wall, as I have dedicated it. I love how it is connected to my full 
knowledge Map, which is available through a few gestures. 

This then illustrates my first wish, to be able to have as many traditional windows 
as I like, and place them where I like, including onto named places, such as a Map wall or 
Photo wall. 

I then have a meeting with Fabien and we appear as talking heads for most of the 
time, so that we do not have to be self-conscious about small movements like reaching across 
for a coffee and so on. When necessary we can enter the space fully, to present something on 
a virtual screen for example. Fabien can see all of my room, as I have given him permission, 


it’s like he is in my physical office. When we work we share documents as though they are 
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physical documents, and just as easily. He asks me to have a look at the VR environment he 
is building and I join him, as easily as walking through a magic portal. 

This sounds pretty simple but there are a lot of aspects needed to make this work. And 
we should make it work. How to share data, how to enter and leave spaces and to what 
degree we choose to be present. 

In the further future I don’t know what I want. But I need to experiment to experience 
to understand. And this can only happen when we have an actively open environment in 


which we develop. 


The Politician and the Headset 


I show the headset to a politician who ‘doesn’t get this VR thing’ and when it’s on her head 
she is immediately struck by how immersive it is. I then present her with her laptop’s screen 
and she experiences what I showed you above; I change the screen, move things into the 
world and have a meeting. She takes her headset off and says this is not what she expected, 
she expected a more of a toy world with Zuckerberg cartoon people. I tell her about the need 
for open development infrastructures and she says the market should decide, this is too hard 
to regulate. I then tell her what she saw was all completely fake for a demo and she takes the 
headset back on and tries the real systems and after a while it becomes clear that the lock in is 
worse than it was with Microsoft Office and sharing information is nearly useless. This is 
where I see politician starting to flicker her eyes in recognition of the what is at stake. 

Of course, I can’t afford to develop such a system to make such a presentation, but can 


we afford the opportunity cost of not pushing for more open development worlds? 
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Space-Time Compression 


Leon van Kammen 


The next step in VR technology, in my opinion, is for it to become hyper-navigable, 
addressable, and shareable. 

This is not a feature that app stores can provide, because they are designed to focus on 
separated, mono-curated, mono-layered experiences. 

If VR operating systems are to be appealing, they need to offer the opposite as well. 

One low-hanging fruit to achieve this is for VR headset manufacturers to enable certain 
multi-layered experiences by default. 

This would allow the app store layer and the web layer to coexist, enabling users to 
access notes on the web while experiencing an app store. 

Even by restricting the web-layer to plain HTML without scripts & rich-media would 
unlock enormous multi-layered potential. 

By doing so, VR/XR device manufacturers can take advantage of innovation waves in 
multiple directions simultaneously, instead of ‘waiting for the next killer app’. 

Additionally, a single solo-button that hides yourself/others should be added as standard 
privacy-equipment, making multi-party experiences always optional aspects of any VR 
experience. 

As most people read books or write emails in solitude, this would open up greater 
opportunities for introspection, cognition, and meditation. 

To complete the circle of text unfolding itself to (VR) experiences and vice versa, we 


need layers that promote these conversions. 


Continues as URL as Space Time Encoder. 


56 


a7 


Writing On The Wall, Text In Virtual Reality 


Mark Bernstein 
a contribution in honor of the fifty-fifth birthday of Frode Hegland 


The writing is on the wall: virtual and/or augmented reality is real. Or, perhaps, it will 


become real. What is the future of text in the virtual world? 


Fiction anticipates virtual screens will be much like physical screens, or the projected data 
surfaces of Starfire [11]. Here’s Arkady Martine’s vision of a day at the office in the heart of 


the galactic empire: 


The interface was...more intuitive than Mahit had expected. She gestured and the 
Infograph responded—spreading her hands and twisting her wrist spawned 
multiple transparent workscreens, and she could make her own halo of 
information. She found Nineteen Adzes preset camera feeds and called up the 
one that was still trained on the demonstration...and set it to run off to her right 
side. Over her left should she put a window full of a running stream of tabloid 
headlines, and resolved to improve her vocabulary of casual and insulting 
vernacular—and perhaps also to learn something more about anti-imperial 


activists. 


Martine Arkady [13] 


Martine’s Empire has planetary cities, cheap interstellar travel, a history spanning thousands 
of years, a culture and politics of Byzantine sophistications, yet their screens look a lot like 
our screens. Indeed, beyond their translucence, they’re not very unlike Murray Leinster’s 
“Logics” of 1946. 


It looks like a vision receiver used to, only it's got keys instead of dials and you 
punch the keys for what you wanna get. It's hooked in to the tank, which has the 
Carson Circuit all fixed up with relays. Say you punch "Station SNAFU" on your 
logic. Relays in the tank take over an’ whatever vision-program SNAFU is 
telecastin’ comes on your logics screen. Or you punch “Sally Hancock s Phone” 
an’ the screen blinks an’ sputters an” you’re hooked up with the logic in her 


house an’ if somebody answers you got a vision-phone connection. But besides 
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that, if you punch for the weather forecast or who won today’s race at Hialeah or 
who was mistress of the White House durin’ Garfield ’s administration or what is 


PDQ and R sellin’ for today, that comes on the screen too. 
Murray Leinster [15] 


This stability of (partially-occluded) vision might reflect our impoverished imagination, but I 


suggest it points to some facets of text that we seldom have occasion to discuss. 


Visual Field 


I have argued for nearly 25 years now that our urgent goal must be to invent forms of writing 
that are “better than books” (Bernstein, 1999). The codex we know does not meet our needs, 
and never has; it is neither divinely ordained nor the end-product of two millennia of 
development. It finds itself , as it always does, at the rapidly-changing intersection of 
numerous economic, intellectual, and social force (Bernstein, 2023b) (Bernstein, 2023a). 

Yet in one respect the contemporary codex comes close to being everything it can be: 
books occupy just about as much of our visual field as we can spare. We know precisely how 
small we can make our letters, because we are deeply interested in some codices that were 
originally conceived not something to be carried in a pocket but as a library to be stored in a 
dedicated room, maintained by a dedicated community (Amory, 1986). If we could make the 
text in the Oxford English Dictionary smaller, we would (Raymond and Tompa, 1988). A 
quarto block of 30# book paper weighs about 0.75 1b/100 pages, and if this book was part of 
the documentation you need to keep on hand aboard a commercial passenger plane, the 
aircraft designer had to anticipate the fuel cost of keeping those pages almost continually 
aloft over a span of 30-50 years (Malcolm et al., 1991). We’ve known since before the 
beginning that we’d like print to be smaller and the page to be bigger, but we simple cannot 
manage that (Buckland, 2006; Bush, 1945). 

The glowing informatic halos of interstellar empire resemble the screens we know 


because they must. 
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A Library Table Of One’s Own 


When I was an undergraduate at a small American liberal arts college, some students each 
appropriated a personal table in the school’s small but under-utilized science library. There, 
we'd array books and notebooks, textbooks and assignments, sources lying open to await our 
return. This was great, but you couldn’t get away with that at Harvard, where even the most 


obscure libraries were far too busy. 


Those library tables, though, might indicate something that virtual reality could give us: 
ready access to assemblages of information we have laid out for use, residing at the edge of 
our vision or conveniently beyond it. Indeed, the familiar map views of VIKI (Marshall and 
Shipman, 1997) and Tinderbox (Bernstein, 2011) seem at least a gesture in this direction, yet 
they are far less informative (and far less comforting) than a table full of references set out 
before us. The book wheel is an oft-cited but seldom-used anticipation of intertextuality; one 
survives in the Strahov Library in Prague. Virtual space might well accommodate 
comprehensible assemblages, laid out for our convenience, to which we might occasionally 
admit a colleague or collaborator. 

Reading multiply is a useful skill: histories from Gibbon (see (Grafton, 1997)) to 
Lekson (Lekson, 2018) may best be approached by reading two copies at once: one open to 
the text and one to the footnotes. 


Written Above The Door 


The sign above Hell’s Gate instructs the viewer to abandon hope. Orpheus ignored it 
(Mitchell, 2020). So did Dante. Quite a lot of what we write on buildings is not meant to be 
read, or at any rate, not meant to be understood. 

Even when we write on walls with the intent to be seen, as in graffiti, it is far from clear 
what, precisely, we wish to express. It mattered to Don Diego de Vargas that he had visited 
El Morro in what he had decided to call New Mexico in 1692. But what reader is envisioned 
here, and what is that reader expected to conclude? 

“The moving finger writes”, Edward Fitzgerald reminded us in Stanza 71 of his 
Rubatyat: the writing on the wall can matter. But from Lascaux’s horses to Mucha’s art 
nouveau actresses, public inscription is often ambiguous, ambivalent, and wayward. The 
other epigraphic tradition, of course, involves things written on stone and displayed in public 
as an deterrent to corruption and special-dealing: anyone can read (say) the Athenian Tribute 
Lists, the Fasti of Rome, or the Laws of Hammurabi (or Moses), so their content is widely 
known. Additions and forgeries are difficult because they’re written in stone in the middle of 


a busy marketplace. 
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This matters to virtual spaces in two ways. First: stability. If writing persists, it marks a 
place not only for us but for everyone else. Conversely, if we know writing will not persist— 
if it will be burnt at the end of the festival or washed away by the tide (like tears in the rain) 


—that instability can inspire memory or reinscription. 


In the winter of 1983, I stood on a barren hillside hundreds of feet above the coast of 
the Black Sea, 
and I promised myself that yes, I would take care to record in my mind, and later 


remember, 
the time that was to come, each and every moment of it. It was a promise I kept well. 


Magdalena Donea, Moments 


Secondly, there does remains an architectural surface on which people inscribe ephemera: the 
chalkboard. The utility of blackboards for instruction has largely been superseded, but in 
mathematics they remain an important resource for working through tentative, inchoate ideas. 
Chalkboards work because they are easily erased, and because their scale is suitable for 
collaboration (Wynne, 2021). It has long been recognized that virtual spaces lend themselves 
to rescaling, and that this could be valuable when working with large sets of equations or 
large sets of data that cannot adequately be represented as a statistical ensemble (Asimov, 
1953). 

Yet, once more, we are left with a virtual representation of the familiar: a plane (or, at 


any rate, a continuously-differentiable surface) subject to simple conformal mapping. 


Architectonic and Gin 


When Bolter described hypertext, he used the term “architectonic writing,” which is to say, 
writing with (or on) structure (Bolter, 1991). The choice of adjective has not inspired much 
comment, though (Kaplan and Moulthrop, 1994) holds up rather well. Bolter’s 1992 
Hypertext keynote discussed hypertextuality in terms of museums and memory palaces, an 
approach that seems well suited to virtual spaces (Hargood et al., 2018). Yet physical spaces 
present significant obstacles and difficulties; the audience can see things too soon, or miss 
things because they’re on the wrong side of the hill or too far from the parking lot (Millard et 
als.2013); 

Why architectonic and not architectural? One reason, of course, is that the term 
“architect” was the focus of a prolonged struggle to establish a profession analogous to 


physicians and attorneys. Anyone who builds can be a builder, a tekton, but only a 
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professional can be admitted to the rank of architect—and only at the call of established 
architects. The point of “information architecture” (Rosenfeld and Morville, 1998) was that 
professional information architects would run things and would rein in the exuberance of 


waywardly unprofitable creatives. The less-familiar term “architectonic” sidesteps this issue. 


More interestingly, “‘architectural” has acquired a vernacular connotation of decoration 
or ornament, as in “architectural millwork” or “architectural lighting”. Architectural writing 
might be writing on the wall. 

From time to time, hypertextualists remind us of Sullivan’s dictate: form ever follows 
function (Sullivan, 1896). We ought to remember here what Sullivan built, at least as much 
as what he wrote. The bank of Owattona, Minnesota did not insist on those chandeliers: 


Sullivan did. It might repay us to contemplate why they are as they are. 


Structure 


Our hypertext maps are, proverbially, boxes and arrows, expressions of structure that strive 
for legibility, and in which the the complexly tangled web view of Intermedia serves as a 
terrible warning (Conklin, 1987). 

All scientists believe that the observed world is structured—that it is not arbitrary or 
random, that is is (at least sometimes) knowable. The hypertextual tradition grows, in part, 
from the early 20th-century effort to formalize knowledge, to set mathematics and philosophy 
on a sound foundation (Budiansky, 2021). That effort revealed that, in fact, not everything 
that is true can be proven, and in doing so established clear limits on what would become 
computer science. 

Everything is intertwingled, and always has been (Nelson, 1974). Not all hypertext 
maps are neat. Our representation of the world ought to be as simple as it can be, but not 
simpler. 

Our visualizations valorize clarity, but clarity is not always to be had. Obscurity might 
indeed be our own failure, but sometimes things are obscure because the moon is down and 
the Pleiades set. If we find ourselves (as we do) on a darkling plain swept with confused 
alarms of struggle and flight, that’s the way it is. It is neither our fault nor the punishment of 
our sin. It is the way the world is. Some things cannot be measured (Heisenberg) and some 
true statements may be unprovable (Gédel); that came as a distressing shock, but we’ve had a 
century now to recover. The world we see — say, in Sargent’s Daughters of Edward Darley 
Boit — is replete with dimness and darkness, lost edges, with faces turned away from us or 
hidden from sight. Many of the demarcations and boundaries in our idea planes might not be 


needed once we accept that perfect clarity cannot be attained. Removing them might well let 
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us see better. Non-Euclidean geometries warp and transform the plane unexpected ways; in 
particular, the hyperbolic space of the Poincaré disk (and the Gaudi roofline) neatly 
accommodates the combinatoric explosion to which taxonomies and tree structures are so 
often inclined. (Bernstein, 2023c) 
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Methods of Al & VR — Stitching the Bell Curve 


Patrick Lichty 


Al is changing the fabric of reality if one considers the totality of amassed texts within 
mediated environments as the fabric of encoded experience. One of the main concerns of the 
coming of AI is outsourcing cognition to synthetic agents. This reflects the shift from the 
spoken to the written story through the allegory of the Egyptian god Theuth and the Greek 
king Thamus.[2012] In return for written language, the price would be that of oral culture or 
the remembered record. However, historical accounts tell of relatively factual information 
retained for generations through oral culture. Even with the transcription of scribes in the 
Middle Ages, there were copious notes in the margins and alliterations between 
transcriptions. The abovementioned issues bring forth two issues, translation, and veracity, 
before even going to notions that Hegland put forth using VR as a cognitive filter for artificial 
intelligence. (2023) 

The first translation issue has different elements yet to be addressed as of June 2023 
(the time of this writing) regarding artificial intelligence engines using LLMs, such as GPT4. 
These errors relate to the relationship between the large language model and the translated 
input accessing it. This relationship, while given base parameters by the program, is defined 
mainly through the training and self-programming of the engine. This process could be 
better, and semantic associations with the translator may not make sense to the human 
operator and are mainly a result of the internal consistencies of the model. 

Secondly, veracity is one of the main issues with machine learning and LLMs. This 
problem is relatively simple and is explicated easily by a query for biographical or 
bibliographic information, with the engine making up several cases, books, or non-extant 
articles. (2023) This is analogous to trusting oneself to a calculator, rather than doing the 
manual calculation, and having the answer for "3+5" equals 7.325, and the human operator 
saying, "Well, that's close enough...." In this writer's assessment, the willingness to accept 
less than accurate representation by technology opens humanity up for nothing less than the 
inaccuracies in shipping logarithms that resulted in imagining Babbage's Difference Engine 
in the first place. (2002) With such an acceptance of unreliable narration for the sake of 
imagination, the process is potentially disastrous. Even if it attributes itself to the complexity 
of the engine, when a lack of veracity is proven, it should not be relied upon. 

The flaws inherent within contemporary artificial intelligence bring us to an initial 
stopping point for the widespread adoption of machine learning and large language models 


concerning their reliability. Microsoft's adoption of artificial intelligence in the Bing engine 


66 


(2023) and elsewhere has already ignored these concerns. Again, this writer contends that 
this is a function of capital's need for early adoption versus the lagging time frames of 
legislation in the nation-state, which is only partially contiguous to infopower. Once this is 
said, the superstructure will likely adopt machine learning before it attains a suitable level of 
veracity/reliability. 

The next issue is, as Hegland states, the inverse bell curve between Artificial 
Intelligence (AI) and Intelligence Augmentation (IA), referring to Engelbart. Engelbart’s 
intelligence augmentation model called for improving computer systems, which would 
challenge the human organism to improve itself to work more effectively within it, creating 
the next evolution in computation. (2001) Artificial intelligence merely takes on more 


cognitive load than "user-friendliness" and other human factors. 


One solution is to change the regime of human factors when managing computation. 
One idea is that of using virtual reality, but having been a user of Microsoft’s Hololens, two 
interface regimes come to mind: spatial computing and Augmented Reality or XR. 

The interface regime of spatial computing, as shown through the Hololens, is scanning 
the local environment and using that as where one can place holograms, virtual desktops, and 
interface components. A critical element to this is being integrated into XR headsets, such as 
the Meta Quest and Apple glasses, which is hand gesture recognition. If a computing regime 
evolves from the keyboard, mouse, and screen to that of virtual representations of 
computational entities, browsers, holograms, or otherwise, a natural interface regime would 
include that of manual control using one's fingers. From a purely HCI perspective, this 
allows for the natural manipulation of digital constructs as natural objects. 

Secondly, and this is in line with the notion of spatial computing, augmented reality is 
superior to virtual reality in that it retains a computational space of the physical environment 
around the user with the addition of virtual contact. From a fundamental perspective, this 
relates merely to keeping the simulated world grounded in that of the quotidian. In a more 
abstract sense, when considering artificial intelligence and managing it in terms of a spatial 
environment, grounding a virtual model in the physical environment is a veracity check. The 
constructs the machine learning models put forward are checked against the potential lack of 
"reality checking" that could emerge in VR. Reality checking concerns virtual environments, 
merging with artificial intelligence and creating infinitely bespoke and aestheticized worlds 
that create significant cognitive gaps between conventional and virtual reality. 

Much of this writer's theory on the importance of grounding virtual and augmented 
realities in some conventional frame relates to Erin Manning's work on embodied thought. 
Much of Manning’s theory of mind relates to the intersubjective dynamics of relational 


movement and ‘bodying,' or the relation of the body to the physical environment. (2014) 
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Cognitive dissonance arises until that body is entirely contiguous to that of the virtual content 
and its relation to the environment. Therefore, this author argues for using augmented or 
extended reality within a regime of spatial computing as the mise en scene of embodying AI 
in Intelligence Augmenting environments. The cognitive load relieved by the information 
synthesis of the AI engine counterbalances with the grounding/checking with the physical 
environment (veracity) and the necessity of intersubjectively 'bodying' with the environment 
and the virtual constructs. In this intersubjectivity, this author argues for the possibility of an 
Engelbartian intelligence-augmenting scenario. 

However, the problem with artificial intelligence remains in its operational flaws, even 
before it can expand into large environments as a structural milieu. However, from a purely 
gestural point, this author agrees with Hegland regarding needing fundamentally different 
interface regimes to incorporate large learning systems. These new systems must exhibit 
robust veracity and allow fluid integration into intersubjective environmental paradigms. In 
this way, AI might be the foundation of creating spatial computing environments that allow 
the human organism to expand within these environments while remaining grounded in 


objective reality. 
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On the Eve of the VR Paradigm Shift 


Matthias Miiller-Prove 


1963: Douglas C. Engelbart writes A Conceptual Framework for the Augmentation of Mans 
Intellect.'° 

Ivan E. Sutherland finishes his thesis Sketchpad - A Man-Machine Graphical Communication 
System!® on Sketchpad, the first interactive CAD system on a TX-2 computer. 


1973: Xerox Alto — an experimental personal computer at Xerox PARC that was used to 
explore and design graphical user interfaces (GUI), local networking (Ethernet), laser 
printing (raster fonts) and object oriented programming (Smalltalk). Alan Kay coins the 


phrase, » The best way to predict the future is to invent it.« 


1983: Apple Lisa!” — second commercial computer with a graphical user interface; after 
Xerox Star in 1981 and the Apple Macintosh in 1984 


1993: The World Wide Web is donated to the public domain. [The only remaining legal copy 
of the certificate is owned by Robert Cailliau. /cf. Interview with Robert Cailliau, 2018!%] 


And 2023? 


History is perceived as a sequence of logical steps only though the rearview mirror. The brief 
compilation of computing events form 1963 onwards is no exception. Nonetheless, it is quite 
likely that 2023 will be seen as a significant year because it marks once again a shift in 
computing paradigms. We went from punch cards and command line interfaces (CLIs) to 
GUIs. Since the 1990s desktop windows coexist!? with browser windows to access the Web. 
Mobile touch devices are part of our daily life since the first decade of the millennium — the 
iPhone was introduced in 2007. 

Paradigm shifts in interaction design do not replace the previous modalities. In fact they 
are all used in parallel today. [cf. Back to Childhood — Infantilisation of UI Design, 20112 
Depending on the tasks, we still enter text commands at terminal shells or search engine’s 
textfields; we drag file icons around, and touch, swipe and pinch pixels behind glass. Voice 
commands basically swap keyboard input with speech recognition. Due to the emulation of a 


human conversation, the interaction feels even more natural — until some commands are not 
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properly interpreted and the illusion falls apart. Users have to cope with it. The systems 
represent decades of development that are layered on top of each other. Usage is only 
intuitive for a very small scope — on a larger scale metaphors and command structures are far 


form coherent and consistent. 


What’s next? 


2023: AI and VR 
The research lab openAI has ignited an industry hype by launching chatGPT 3.5 last year and 


chatGPT 4 in 2023. Large language models and deep artificial neural networks generate text 
at a level of quality that seemed to be reserved for human abilities until now. By harvesting 
an average of all human writing, AI systems get better to “understand” the intention of their 
users and to respond in a seemingly reasonable and correct fashion. However, algorithmic 
philosophy does not yet provide a framework to describe what is happening in this area. Vint 
Cerf put it this way [Future of Text and AI, 20-Apr-20232!]: »Our problem is that the Turing 
test turned out to be too easy now for the ChatGPT to pass.« It might be too early to provide 
an assessment on the techno-social implications of the new computing capabilities. 

The other technique that eventually reaches the professional and prosumer market in 
2023 is VR/MR/AR (virtual, mixed and augmented reality; cross reality —XR- is used as an 
umbrella term.) XR systems are under development for quite some time. A year before 1963, 
Morton Heilig has introduced the analog system Sensorama, that enhanced a movie clip with 


stereo-audio, aromas, wind, and vibrations. [interview footage with M.Heilig?”] 


The Sword of Damocles?? by Ivan Sutherland et al. (1966) is considered as the first 
computer-based MR prototype: The user was able to inspect a free-floating cube by moving 
around. His perspective was tracked with telescopic arms from the ceiling, hence the nick 
name. Since then various research and development teams aim to provide a satisfying user 
experience in virtual worlds for play and work. Google Glass, Microsoft HoloLens, and 
Oculus Quest to enter the Metaverse are recent products. 

For decades Apple has a track record to shape the industry by its products and services. 
The desktop publishing revolution was fuelled by the Mac. The digital music industry was 
completely disrupted by iPods/iTunes which lead to the iPhone and the AppStore market 
place. Other companies like google, Samsung and Microsoft followed to build smart phones 
and tablets and their own operating systems and app ecosystems. 

Tomorrow, on June-5, 2023, Apple will introduce its VR headset. Like in desktop-GUI 
computing or mobile touch devices, all other research and development teams will conduct 


competitive analyses on Apple’s design solutions. This will inspire and inform their own road 
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maps. All together they will establish a de-facto standard how to interact in XR environments 


by means of head-mounted-displays, gestures and plenty of sensors. 


As mentioned before a paradigm shift in human computer interaction has always just 
added a new layer on top of the established stack of HCI design patterns. Users will continue 
to choose the device and the input mode that best suits their tasks. As a consequence this 
means that data, tools and users will and shall be able to seamlessly migrate between devices 
and systems. 

A user — Eve — might start to take notes on a new idea by recording a voice message to 
herself. Later she opens the list in text-form on her laptop computer and continues to refine 
her concept. In order to incorporate related research and references she might virtually dive 
into the ocean of literature and other media of her field. The 3D info space is arranged in a 
way to make her feel comfortable and safe. It enables her to discover related materials with 
ease. Connecting the dots, drawing conclusions, and enhancing her considerations is a 
computer-augmented thought process. 

It is a desired future to reuse and further improve common interaction modalities for 


XR — if possible and if it makes sense. Let’s take a look at the established interaction areas. 


The desktop metaphor is fundamental to personal computing. For virtual and 
augmented scenarios it has to be replaced by a 3D rooms metaphor. Be reminded, that 
metaphors in computing are not just digital copies of real objects; instead they use familiar 
concepts and add extend the conceptual model to offer functionalities that would otherwise 
not be understandable and usable at all. 

WIMP (an acronym for windows, icons, menus, pointing device): windows, i.e. 
rectangular areas with scrollbars to reveal sections of a much larger canvas... Windows can 
overlap like sheets of paper on a desk. Due to the limited screen real estate mobile devices 
don’t have windows Here, the only window is identical with the screen. Virtual rooms do not 
have edges — everything is projected into a sphere. Stereoscopic sight adds the third 
dimension of vicinity and distance. It has yet to be decided if metaphorical shelves —like the 


stockroom in The Matrix?+— offer good access to documents and media files. 


Icons either represent (closed) documents and folders or they provide cues for 
command buttons or apps. Both aspects are needed in XR. But simply using the graphical 
language of GUIs and mobile devices for VR seems not to be appropriate. New symbolic 
items that are born-3D should be developed and evaluated. 

The use of menus is slower than other command panels like e.g. tool bars or hotkeys. The 
primary benefit of menus is a complete overview on the functional space of an applications. 
The lack of a standard menu structure for apps on mobile devices makes it extremely difficult 


to occasionally use apps. Apart from a few standard buttons there is no familiarity with apps 
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that are used infrequently. Obscure interaction gestures remain hidden and cannot be learned 
by climbing through the menu tree. XR will face the same issues until standards and 
conventions will help the users to feel safe and in control in new XR app environments. 

The P in WIMP stands for pointing device. Originally it was a mouse, but today 
trackpads are more common than mice for laptop computers. Another advantage of trackpads 
is multi-finger gesture support. These gestures have been adopted from mobile devices. 
Pointing and multi-finger gestures will also be used for XR interactions. New gestures will 
involve the hand, both hands, and even body gestures like nodding or shrugging the 
shoulders. Some of these new gesture can eventually even be used on other device categories. 

A unique UI element that can be triggered by point and click is the hyperlink. It 
connects two Web pages with each other and teleports the user from a marked piece of text to 
a destination somewhere else on the Web. Magic portals are already used in VR games to 
change levels or virtual scenes quickly. Generic XR hyperlinks might connect various XR 
information environments with each other much like a browser jumps from one webpage to 
another. 

VUIs — voice user interface — is the next interaction modality that evolved from speech- 
to-text dictation systems into voice command systems for mobile, car and home 
entertainment and shopping devices. As soon as the error rate comes down to an acceptable 


niveau, voice commands can be an additional input modality for XR environments as well. 


The blinking i-beam cursor has been invented in the late 1960s. 


Larry Tesler introduced text selecting by dragging the mouse and cut/copy/paste 
commands for the clipboard metaphor at Xerox PARC in the 1970s. Innovation research 
shows that new paradigms have a lead-time of at least a decade and also a life-time of half a 
century and more. This can be explained with The Pace Layer Model by Steward Brand 
[introduced in his book The Clock Of The Long Now: Time and Responsibility, 1999]. 
Changes in infrastructure are much slower than market cycles of products or the fashion 
collections of the seasons. Media theory says that a new medium will first use and embrace 
the former medium before a new appropriate language for its content formats will evolve. 

XR is in the same situation. Prevailing GUI and gesture/touch and VUI interaction 
design systems will be first adopted for XR before a mainstream of typical interaction 
behaviours gets established. Chances are high that 2023 will be seen as the year of an AI 
break through and of an paradigm shift for XR due to the launch of Apple’s headset. 


) 


It is important to consider the entire suite of devices and to design for scenarios that 
involve the flawless migration between devices, various applications, apps and virtual 
environments. Proprietary silos will slow down innovation. Generic and flexible interaction 
methods, that are always available like the clipboard, will improve the adoption of XR 
environments and increase the joy of use. Standard system behaviour will flatten the learning 


curves an improve productivity. 
Now is the time to shape the XR tools in order to support and augment collective human 
reasoning and collaboration. 

The user Eve will decide whether Apple’s solution fits her needs and matches her 
expectations regarding augmented and virtual reality — or whether the industry needs to 


come up with even better tools to enhance Eve’s digital life. 
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Textual Affordances in VR & XR 


Peter J. Wasilko 


We stand on the cusp of a new era in Human-Computer Interaction. In the days and weeks 
ahead a number of new immersive display platforms will be entering the market with new 
headsets arriving from Meta [16] and Apple [17], with rumors of a Samsung, Google, and 
Qualcomm XR Headset collaboration [18] and hits of new Xbox VR hardware [19] in the 


mix. 


Interaction Modes 


Rather than speculating on the specifics of any given current or emerging offering, we can 
abstract away all the details and think in terms of operating modes, some or all of which 


might be supported in any given hardware configuration. 


Full VR 

In a full VR mode, all external audio and visual stimulation would be blocked by the 
hardware to present a user with a fully immersive virtual environment which might take the 
form of a totally abstract data space [20], a freely navigable semi-realistically to photo- 
realistically rendered 3-D Virtual World like SineSpac [21], or a passive 3-D Film or 
navigable set of linked 3-D Photo Nodes in the QuickTime VR / Google Street Views 


tradition. 


Basic AR 

In a basic AR Mode, one’s device either consists of clear lenses off which an auxiliary Heads- 
Up-Display image is projected (rather like the classic Pepper’s Ghost stage illusion) or of a 
traditional VR headset augmented with high resolution pass-through video cameras 
presenting an image of the outside world composited with computer generated imagery to 
superimpose virtual screens on one’s environment. Such actual or virtual HUDs can either 
track one’s head motions to appear fixed relative to it or be dynamically resized and 
reoriented in response to head movements so as to appear to be stably anchored at a fixed 


position in 3-D Space around which the user can freely move. 
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Mixed/Extended XR Mode 
Finally, in an XR Mode, one’d headset is further augmented with LIDAR or other advanced 


sensors allowing the device to generate a 3-D model of the space around it, making it 
possible for the system to identify real world surfaces and planes onto which 3-D Models can 
be overlaid so one might be able to replicate the appearance of the Holographic Chess game 
in Star Wars: A New Hope or to seamlessly replace a wall poster present in real world or 
create the illusion of a real live kitten frolicking on one’s sofa that cannot be visually or 
audibly distinguished from a real pet until the headset is removed. In theory, future 
generations of immersive technology may eventually employ room-sized holographic, light- 
field, or auto-stereoscopic displays; or give way to neural chip implants to obviate the need to 


wear a headset. 


Adadressability 


With this broad range of possibilities in mind, let us now turn to the place that Text will have 
in such environments, but first we need to briefly touch upon the critical concept of 
Addressability. 


Spatial Addressability 
First, we need a way to name the Space we are dealing with, which can either be the Real 
World, some arbitrary Benediktine Cyberspace , a small local simulation like The Construct 


in The Matrix, or a massively multiplayer online role playing game universe, like Eve Online. 


Temporal Addressability 

We then need to specify a Time Index which can be an absolute point or range in time at any 
given granularity of concern (e.g. a given year or a specific date), a relative temporal offset 
(e.g. “531 milliseconds’ into a named experiment, ‘5 to 7 seconds’ of a given YouTube video, 


or ’20 minutes after rebooting’), or cyclic (e.g. ‘every third Tuesday’, ‘in the winter’, or ‘after 
dark’). 


Granularity and Naming 

Similarly, we need to be able to specify regions / sub-spaces and point locations which might 
be given in absolute geospatial global coordinates, as a named geographic region with known 
mapping to such coordinates, or in a relative fashion with reference to a model or model class 
like ‘the kitchen’, through an explicit naming of a 3-D Cursor location, or a spoken or typed 


search query capable of returning multiple results as a list abstraction. 


a 


The Complete Viewspec 


Beyond simple search queries we can also assume that various filters and modalities might be 
employed by the system whose parameters would also need to be tracked along with a 
Viewpoint and Orientation within or relative to the space or point under consideration. Only 
with this full range of data would we be able to specify a Viewspec [22]with sufficient 
information to capture what we are seeing in VR. Note that we are contemplating the ability 
to replicate a navigable immersive experience in toto rather than making a 2-D screenshot of 
the virtual world or creating a static instantaneous 3-D Model of that portion of the VR World 


visible to the user at the moment of requesting the viewspec capture. 


Implementation Possibilities with IPFS and Mnemonic ID’s 
In order to achieve this level of full addressability and fidelity, we are essentially 
contemplating the sharing of the full VR Model + Viewspec + data-streams employed by the 
system at the moment of capture. Clearly, this is too much data to be packed into a feasibly 
exchangeable identifier, so the model and data stream would no doubt be incorporated by 
reference to pre-packaged physical media or networked resources. Even so, the balance of the 
viewspec would be too beefy to fit in XR Fragment [23], but we can get around this by 
putting the full viewspec and associated data-stores on the IPFS under a Humane Mnemonic 
Identifier like “Purple Elk on Mt. Fuji” [24]. 

Assuming now that we have a way to select objects and spaces in our virtual 
environment and that we can associate them with Mnemonic ID’s like “Green Crane under 
Willow” we can think about how to work with text in VR. 


Severability and Transferability 


Some years back I proposed a Principle of Severability [25] arguing that users would be 
unwilling to invest extensively in creating user-generated-content within an any given 
environment unless they were provided with a mechanism to export that content to modify it 
or in a worst case repurpose it should the platform fail. 

Today, I would like to make a stronger claim for a Principle of Transferability and argue 
that for Texts to achieve their maximal utility ina VR/XR World they must be fluidly 
transferable within and across platforms. The closest example of such a capability is offered 
by the MacOS Handoff facility which synchronizes the clipboard and some application state 
across devices sharing an AppleID iCloud login within the Apple Ecosystem. But this level of 
Limited Transferability falls short of supporting an XR Environment that might contain fully 
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virtual monitors, representations of Windows, Linux, Mac, and home-brew based devices, as 


well as avatars of participants using multiple-brands of headsets. 


Stephenson’s Hypercard 

What we need is something more akin of the Hypercard of Neal Stephenson’s Snow Crash 
[26] which served as an avatar for an arbitrarily large and complex chunk of data. In short, 
Stephenson’s Hypercard is a virtual artifact represented visually as a traditional business card 
holding the address of a datastore that could be copied via a handing off gesture to provide 


the recipient with copy-on-write access to that data. 


Pick and Drop 
Clearly we need a mechanism to transfer text and other data across devices that aren’t 
necessarily owned by the same person unlike the current case with Apple’s Handoff 


technology which doesn’t permit handing off between multiple users. Sony’s Jun Rekimoto 


anticipated this concern in the late 90’s by proposing a pen-based mechanism called Pick and 


I'm transferring computer 
data through the network. 
° 


Drop [27]. 


I'm physically picking up 
a computer object with 


Figure 2: The conceptual difference between remote copy and Pick-and-Drop 


This system would employ a “pen” artifact that could be correlated with a data object on 
screen, such that the Pen’s PenID would be bound to a network data transfer specification 
allowing a receiving device to automatically initiate a background network file transfer of the 
associated data object. Rekimoto’s system was somewhat limited by the need to construct 


pens whose physical proximity and ID could be detected by target systems. 
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Toward Tangible Hypercards 

In VR we can combine Stephenson’s and Rekimoto’s insights to envision a cheap deck of 
laminated playing cards bearing a Mnemonic ID on one face and a QRCode on the opposite 
face. When produced in an XR environment the System would use machine vision to register 
the QRCode and track its position in World. We could then support a familiar ID Badge 
Swipe gesture waving the card over a textual selection or a real or virtual object present in the 
environment to copy it onto the card, this could cause the system to simulate a color change 
to indicate its loaded status. Tapping the card against a data reception target in the XR 
environment would effect a data transfer. In addition to tracking hypercards, the system 
would also be able to register and subsequently track tablets, phones, and laptops by having 
them display a QR Code or post their GPS coordinates. This would permit the environment to 
orchestrate the transfer of data objects like large texts and corpora across physical devices in 
the real world. Moreover once bound to a hypercard, data could be transferred outside the XR 
Environment by carrying the physical card to some other networked device where its QR 
Code could be scanned or by carrying its Mnemonic ID in one’s head to be shared from 


human memory! 


Wands and Crystal Balls 


Where the Hypercard is an avatar for a chunk of data we can also imagine representations of 
data structures and programs. The most natural representations here would be the Wand as a 
stand-in for a Deque (double-ended queue read “‘deck’’) / clipboard data structure and the 
Crystal Ball as a representation for programs. The wand could be a simple black rod whose 
orientation could be distinguished by cap color or width to allow the XR environment to 
visually distinguish a canonical front and back end. 

Tapping a data transmitting object with either end of the wand would push one item per 
tap onto that end of the wand, tapping a data receiving object would draw off one item per tap 
until the receiving object no longer required additional data. Crystal Balls could represent 
programs that might display or alter some or all of a wands contents. This would serve as an 
embodiment of a Forth like programming language and wands would not need ID’s per se as 


they would be identified by their users. 


VR/XR with a 3-D Mouse 


For sit down VR/XR sessions entailing extensive typing of original text, a 3-D Mouse like 
3Dconnexion’s SpaceMouse® Wireless used off-left of one’s keyboard in conjunction with a 


conventional mouse off-right will likely provide a more ergonomic solution than handheld 
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controllers. The trick will be to provide a Snap-To facility to allow a user to make a fast 
semantic selection of the nearest contextually appropriate element in the 3-D environment. 
This sort of interaction would fit in perfectly with the sorts of abstract textual visualizations 
explored at the MIT Media lab by David Small whose magna opus Rethinking the Book 
[28]is a must read. See also: [29] [30] [31] 


Purely Gestural Techniques 


Since Mnemonic ID’s can obviate the need to carry around jump drives and scroll url’s on 
the backs of napkins, and since the next generation of headsets will no doubt support noise 
canceling earphones, speech recognition, and high resolution hand tracking, we can 


contemplate a range of approaches that don’t depend on secondary input gadgets. 


Frame and Dictate 

The first and most critical element to leveraging text in VR is to provide a mechanism for 
rapid text entry. While nothing is likely faster than a chorded Stenography Keyboard or at a 
slower pace a conventional keyboard for a Sit Down work session; when standing, a Frame 
and Dictate methodology would be ideal. Essentially, the user would first create a virtual 
monitor by lifting both arms and touching both pointer fingers together at the top center point 
on the desired monitor and then sweeping her finger tips out the sides, then down, and then 
back together to trace out of the frame of the desired monitor which would then fade in to fill 
the screen with a blinking cursor. The left third of the virtual display would then being to fill 
in with a speech to text transcript of anything being spoken in the room while noise 
cancelation would block those sounds. The user would then dictate text directed to the screen 
while the noise cancelation tech prevent other users from hearing what was being dictated. A 
snap of the fingers would toggle between dictation mode and conversational mode to allow 
one to alternative between dictation and being heard by colleagues until a double click would 


terminate the dictation session. 


Frame and Flatten / Swipe to Select 

Another key affordance is the ability to detect and capture text in the environment. For this a 
Frame and Flatten gesture could be provided where one would raise one’s hand with thumbs 
and fingers up at 90 degrees to form a virtual text capture frame that could be scaled, and 
spun in 3-space by changing the spacing and orientation of one’s hands. Once in a workable 
scale and orientation a two finger swipe could be employed to select and highlight or capture 


text. 
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Ant Man Scaling Effects 


In a truly advanced XR environment, pass-though video would be replaced with real time 
rendering of a Lidar generated model of the space. This would make it possible to play with 
an Avatar’s Scale by reaching out and grabbing mid air to expand or contract one’s scale 
within the space by pulling or pushing the space apart, (i.e. grab in the air and pull your hands 
together to grow your avatar and shrink the space or push your hands apart to shrink your 
avatar and zoom into the space). This affordance would play very well with abstract data 
visualization, but could also help trouble shoot industrial systems where the XR model might 
be augmented with real world telemetry data and textual documentation panes. While 
zooming in or out, a synthetic shadow in the room could indicate one’s physical location, 
while one’s avatar would appear to grow or shrink by the zoom factor it was operating under. 
Imagine several lawyers studying a contract with one shrinking to mini-me size while 


studying some small print text. 


Tangible Citations 

A natural extension of this sort of visualization would be a document room with multiple 
virtual displays each holding a different legal case or virtual representation of a tangible piece 
of evidence. Implicit in this document collection would be a large number of linkages which, 
if simultaneously visualized, would render the data-space incomprehensible. However, in 
VR/XR we could manifest these links as glowing threads connecting text selections across 
virtual documents only when an endpoint was actively selected. To get a feel for how 
powerful this technique can be in 2-D, play around with the Map view of a series of related 
definitions in Frode Hegland’s Author. 


Implode to Text Effects 


Another closely related use-case might be standing a library or high information density 
environment with lots of things at different scale competing for one’s attention. Here an 
Implode to Text Effect would crossfade from XR to Pure VR fading out everything except the 
text in one’s field of view while animating, rotating, and warping each textual surface to 
bring them into position at a uniform scale in a single space filling two-dimensional layout on 
an ergonomic arc shaped overview panel. (This would also normalize the orientation of 
perpendicular blocks of text on the spine of a book.) The closest cinematic depiction of this 
sort of effect can be seen in some of the visual sequences depicting the operation of Cerebro 
in the X-Men film franchise. 
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Pluck and Plop 


Finally at a more prosaic and minimalist level we could employ a gesture based analog to 
Pick and Drop to transfer text between physical devices and user interface elements. This 
basic Pluck and Plop gesture would consist of reaching forward into 3-D space spreading and 
closing one’s thumb and pointer finger as if plucking a string to symbolize the desire to 
capture the nearest textual or object referent in line with the motion. Where that reference is a 
text pane with some previously selected text the motion would capture both the selection and 
Visual Meta reflecting its source and authorship. Reversing the motion as if tossing a pea 
held between one’s thumb and forefinger would send the data object outward in 3-D space 
until it plopped on the nearest drop target which could be an XR-Representation of an actual 
device; a real or virtual Hypercard; another Avatar who could catch it with a corresponding 
Plucking gesture, in the absence of which, a link to the data would be sent to that user via 
email; a Wand, Crystal Ball or similar virtual artifact; or even oneself to email oneself a 


download link to access the data outside of the system. 


Viewspec Reification — Clap to Capture 


Just as important as it is to be able to capture text, selections, and highlights, we need to be 
able to preserve the State of the Experience. For example in an XR Enabled library, one 
might have employed a sophisticated voice query to highlight books on a given topic that had 
been cited by one of seven authors of interest. Since there is no obvious object, real or virtual 
within the scene to represent this Viewspec, we need a way to reify it into a manipulable 
form. This could be achieved by a Clapping Gesture, which would cause a glowing ball with 
a Mnemonic ID call-out caption to be manifested between one’s hands that would hold the 


viewspec and be subject to Pluck and Plop data transfer. 
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Summary 


Regardless of which, if any, of the aforementioned possibilities or variants there of, that see 
actual implementation, we need to find ways to fluidly enter large amounts of text into VR & 
XR and to move them across vendor platforms. Moreover, abstract text and data 
manipulation functions like key word highlighting, named entity extraction, parsing and 
formatting, that aren’t inherently tied to the mechanics of any given platform need to be 
Severable and Transferable across systems so that they won’t be subject to Walled Garden 
Capture. Only then can we see the same level of Openness and Interoperability emerge for 
VR and XR that made the World Wide Web the Civilizational Change Agent that it has 


become. 
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Cross hatched 
Phil Gooch 


“Will you take me there 
To a distant place I’ve never been before? 
I could leave this world 
I could follow you like oceans to the shore 
You could take me there 
Make the rivers of my mind flow to my dreams” 


Nitin Sawhney, Jmmigrant2> 
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Ironic isn’t it? We exist in four-dimensional spacetime, but we increasingly lived our lives on 
a two-dimensional plane. Heads down, swiping, tapping, gazing. Recording live events 


through our phone screens, rather than experiencing the moment. 


Before, when we used to laugh at those early bricks, we couldn’t imagine then how 
these devices would evolve and shape us. How we’d create new vocabularies, new 


connections, new roles. And yet they would limit us. We stopped looking up. 


Later, we chuckled at the strange headgear, the odd gestures. Like silent disco, it was fun for 
a while. But we were still stuck in these experiential silos. 

In The City and The City [32], China Miévillegave us a glimpse of what was to come. 
While the citizens of BesZel and Ul Qoma had to learn to un-see (‘When in Beszel, see 
BesZel’), we embraced the cross-hatching of boundless realities. Just as three-dimensional 


space stacks an infinity of two-dimensional planes, so we learned to tesseract worlds. 


Remember the big reveal? Those extended earbuds that looked like nothing special. But the 
holograms they projected amazed us. The silos were gone, we could overlay new realities 
seamlessly. We could issue commands by thought, and the AI would turn these into new 
environments. There was no ‘virtual reality’, instead, we could create multiple actualities. 
And not just on Earth. 

Remember the first trip to Mars? We’d had the robots up there for a while, they had 
built the tools they needed to reproduce themselves and 3-D print the structures. But making 
the physical journey ourselves still eluded us. Instead, we used the augmented perception 
technology we had here to transmit and connect our senses with the machines there. Sure, the 
ten-minute time delay is a bit of a pain - but that’s a temporary limitation. 

We may never be able to send matter through a wormhole, but sending information might just 
be possible [33]. 
If all our sensory information can be encoded in bytes, what really makes us human? I 


guess that’s a question for another time ... 
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Versality, Chapter 2.09 


Rollo Carpenter 


Though possibly it was ill-advised, Emi was flattered to be invited to meet Xeen-Xeenic, in a 
non.life space to be opened the next day — just the two of them, ‘on camera’. A very effective 
promo for the space-designers it no doubt would be too. 

“How are you on a surfboard?” asked Xeen in-ear, while Emi was at work. You was on 
the asterdroid Targ, in the middle of attempting and largely failing to make humorous 
conversation with a gaggle of low, squelchy beings around a sulphurous pool. 

“Surfing? Well, you know, I’ve been in the sim-spaces with friends several times... but 


they’re never quite right.” 


“This place is different. Will send the link.” 


Space: NL-ALT1933 SEA-SPACE1.0 
Time: 2098-06-18 15:15:15 gmt 


Once blipped, there they were, barefoot in full wetsuits, on a warm broad curving sandy 
beach, boards in hand, pointing skywards. Not quite a normal beach — not in geometry, scale 
or light. Somehow, without quite seeming too overwhelming, behind them stretched most of 
the African continent, with the Elevator visible on the horizon. A glow beyond, within, lit all 
of it, but the sky was a deep moonless night, stars and Milky Way in brilliant detail, closer 
and more vivid than life, and rotating visibly fast, as the small world turned. 

With their backs to the sea, it was almost above them. As they turned to the sea, now 
the land was above them, and the sea an approachable surface, full of dramatic glowing 
waves. The water was as if in sunny daylight, and yet, magically semi-transparent, the 
strongest of stars shining right through it, well below the ocean’s peculiarly infinite and non- 
curving horizon. White crests atop the waves were spacily — almost darkly — translucent. 

“This looks pretty fantastic, Xeen. So we surf here, now?” 


“Ooh no, Emi, we go places. Surf places. You’!l see! First we have to get out through 
the breakers.” 


Boards lowered to their sides, they ran into the waves. As they paddled themselves out, 
dipping under crests, Emi called excitedly to Xeen, close by, “This is no normal non.life 
water. This is the real thing!” 


“Yes. Problem now solved. But look behind for a moment.” 
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As they moved away from it, the beach, Africa — the world was becoming a globe. Soon 
enough, it was complete — a jewel-like Earthly sphere was floating there in the broad ocean, 
two fifths still slightly visible beneath the surface, turning. The beautiful night sky above 
them no longer appeared to be moving. They were in the deep — in deep sea space. 

Xeen spoke again, perfectly clearly despite the waves regularly crashing over them, 
“The creators of this space probably expect everyone to go outwards first, via Mars and 
especially Jupiter. So let’s go inwards.” Xeen selected your intentions, in-eye. 

The sea adjusted beneath them, and the real waves came, luminously, in the day-night. 
At the third attempt Emi and Xeen both caught their first, only a few metres apart from one 
another. This was no mere coincidence. They did not only ride the waves. The waves rode 
them. 

While ensuring it was never too easy, this space could understand human actions and 
micro-configure the water to ‘happen to’ arrive just right to ensure that even whole groups of 
friends stayed close enough together, caught pockets and wiped out at nearly the same times. 

Xeen knew the Sea-Space was built for maximum visual impact too. If they aimed to 
surf straight to Mercury, Venus was bound to be conveniently arranged to pass close by. In 
the event, it came more than close. 

Just as Emi and Xeen whooped with joy at catching the second of a set of waves, and 
were travelling at what to scale must be near light speed, the hazy light of Venus loomed up 
rapidly on their right. 

“Let’s try not to collide with that thing, eh Emi?” Xeen said. 

Emi raised her eyebrows and smiled lopsidedly at Xeen for a second, but did not have 
much of a chance to respond to the jibe. The invented, hyperbolic orbit of Venus sent it 
sweeping rapidly by, right in front of them. It cut a dramatic curving path through the waves, 
creating complex disruptions in the surf, arriving at drastically unexpected angles. Wipeout. 

Yet, a mere five caught waves later, they approached Mercury, elated but already 
grateful for a new gentleness of the water, carrying them easily towards its ‘shore’ as the 
whole planet distorted in their vision, down, around — from sphere to surface. 

They raced out of the waves, finding Mercury unmysteriously hot and hard to their 
barefoot touch, laughing with exhilaration and at the sheer absurdity of it all. 

“Next stop, the Sun!” shouted Xeen. 

“But the sea will boil,” laughed Emi. 

They could have walked all the way around Mercury, but before long jumped right back 
in. The sea did boil. Xeen and Emi landed ‘on’ the tumultuous Sun in huge clouds of steam, 


and with great shrieks — but their wetsuits, skin and eyes did prove, thankfully, suitably 
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resilient. 


“Don’t tell me... onwards to a hypothetical interpretation of Gliese 667C and its 
planets!” Emi said, humorously hopping from foot to foot, in a pointless yet seemingly 
unavoidable attempt to improve matters a fraction. 

“Why, Emi... how did you guess?” Xeen replied, voice raised above the noise of a 
nearby erupting solar flare, while gesturing open an intensely-glowing screen, to make their 
more complex, abnormal selection. 

They paddled away from the surface of the Sun, now a vast brilliant globe, towering 
above them, filling most of the sky, floating there in the infinite ocean universe. For a 
moment the heavens rotated, all for them, so as to be on the correct axis for the destination 
star. 

The first wave, which was approaching to transport Emi and Xeen unimaginable 
interstellar distances, at imaginarily many times the speed of light, arrived, so tall that it 
almost engulfed the Sun. Within seconds of catching it, the Sun was Sol — far behind them, 


and just another unremarkable point of light in the sky. 
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‘a little doodle as a card” 


Sam Winston 
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e PART 2 


(Apple ‘Vision Pro’ announced, Meta ‘Quest 3’ Released) 


This second section features articles submitted for the Future of Text Symposium after the 
announcement of Apple’s Vision Pro (not ‘Reality Pro’ as the rumours would have it) and 


release of Meta’s Quest 3 which is receiving praise for the quality of display it offers. 


The designs might seem a bit futuristic but soon this will be ancient history. They do at least 
convey the notion of being ‘thinking caps’ to be used when you want a wider and deeper 


view of your work, not necessarily something to work in all day. 


Vision Pro 
¢ Intro video: https://youtu.be/TX9qSaGXFyg?si=kXsP3p68NG5STOz-j 


¢ Information: https://www.apple.com/apple-vision-pro 
Quest 3 


¢ Intro video: https://youtu.be/Exu7r2vZpcw?si=doJP XIV 1CMd2JH7u 
¢ Information: https://www.meta.com/gb/quest/quest-3/ 
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Apple Vision Pro 


Meta Quest 3 
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Leaning into a Novel Take on the Ideal State of 


our Relationship with Al 


Daveed Benjamin 


I first learned about ChatGPT on Twitter after its launch in late 2022. I had been using 
Midjourney for a few months, which got me interested. I was immediately fascinated and 
started using ChatGPT for various tasks. 

After a few days of using the chatbot, I sent an email to the Bridgit DAO list and shared 
posts on my social media, recommending that people try ChatGPT to see how it could help 
them and boost their productivity. One of my early experiments with the chatbot was writing 
a book. 

In just one week, I wrote and edited a book I titled "Dead Web: The Dark Side of 
Generative Tech." The book argued that generative AI might flood the internet with fake 
content, supporting the Dead Internet Theory. However, the "Dead Web" book is on hold for 
now, waiting for more data and developments in the field. 

Shortly after that, I used the chatbot to improve chapter transitions while editing Bridgit 
DAO's book, The Metaweb: The Next Level of the Internet. 
https://www.routledge.com/The-Metaweb-The-Next-Level-of-the-Internet/DAO/p/book/ 
9781032125527 

Now, ten months later, I primarily use ChatGPT and occasionally Anthropic's Claude 
for ideation, writing, editing, and decision-making. I often ask these bots to summarize or 
explain online content. 

During the summer, I heard someone say that in five years, everyone would be able to 
code because of AI. The idea was that AI and no-code tools would come together, making it 
possible for anyone to create a prototype. Inspired by this, I had the idea for a Metaweb 
project on Bitcoin called Ordweb. As an early tech adopter, I decided to build the prototype 
myself with the support of ChatGPT and Cody (an AI coding assistant that knows your entire 


codebase). It's progressing well. 


Should We Pause or Should We Flow? 


Six months ago, in March, when generative AI was gaining momentum, around thirty-three 
thousand concerned citizens, including prominent figures like Elon Musk, Apple co-founder 


Steve Wozniak, author Yuval Noah Harari, and Center for Humane Technology ED Tristan 
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Harris, signed an open letter. This letter called for a six-month pause on AI research. The 
open letter, titled "Pause Giant AI Experiments," aimed to address the growing concern of AI 


systems surpassing human intelligence and the associated risks. 


The questions raised by the letter were significant: 
1. | Should we allow machines to saturate our information channels with propaganda 
and falsehoods? 
2. Should we automate all jobs, even the fulfilling ones? 
3. Should we create non-human minds that could eventually outnumber, outsmart, 
and replace us? 


4. Should we risk losing control of our civilization? 


The letter urged AI labs to halt the training of AI systems surpassing the capabilities of GPT-4 
for at least six months. During this pause, it called on AI labs and experts to establish robust 
safety measures, collaborate with policymakers for effective AI governance, and prioritize 
transparency, safety, and trustworthiness in existing AI technologies. 

At that time, there were many concerns about AI, including the fear that AI might 
become conscious, surpass human intelligence, render humans obsolete, create autonomous 
weapon systems, facilitate online scams, enable students to use AI for essays, perpetuate 
systemic biases, and disrupt the job market. In a WhatsApp group dedicated to shaping the 
future with AI, Christopher Lawrence, the founder of Gatherverse, argued strongly that it was 


not the time to pause but rather to lean in. 


What Might it Mean to Lean Into Al? 


While AI offers significant opportunities for progress, it also raises concerns about losing 
control over its use. For example, the development of autonomous weapons and the potential 
for AI takeover pose significant threats to our safety. It's easy to imagine how these threats 
could lead to dire consequences, including the potential extinction of humanity. Like the film 
“The Terminator.” 

Moreover, there's the risk of losing control over various aspects of our lives, such as 
jobs, economic stability, privacy, and cognitive freedom. We must also be mindful of the 
unintended consequences that can arise from developing AI systems without considering their 


broader implications. 


On the flip side, embracing AI can create opportunities to build safe and beneficial AI 
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systems that help us address these challenges. AI played a role in creating some of our 
problems (like governing search, content, and advertising on internet platforms as well 
malevolent bots and online scams), and it can play a role in solving them. However, we must 
do so thoughtfully and carefully. 


We should also consider the potential loss of our humanity as we continue to develop 
AI. While AI excels in certain areas, there are essential human qualities like intuition, 
creativity, pattern recognition, and problem-solving that may not be easily replicated. We 


must ensure that our technological advancements do not come at the cost of our humanity. 


The Crossroads 


Humanity stands at a critical crossroads. One path leads to democracy, decentralization, 
increased consciousness, self-awareness, and self-determination. The other path, which seems 
increasingly likely, leads to an authoritarian, dystopian future characterized by censorship, 
surveillance, data exploitation, and a lack of privacy. Tech giants primarily use AI for the 
latter, creating a constructed reality that appears to offer cognitive freedom but doesn't truly 
deliver it. This also leads to significant workforce disruption, with many people relying on 
Universal Basic Income in exchange for their data. 

I have a theory. I'll call it “Highest Use Theory.’ Humanity has evolved alongside 
technology, creating powerful tools. These tools can be considered gifts from the creator. As 
our tools become more complex and powerful, we must use them to their full potential to 


fulfill humanity's promise and honor the creator's gift. 


For instance, if I want to run without injury, I must ensure that my running shoes are 
securely fastened to my feet. Similarly, if we desire a democratic future, we must use our 
most important tools, like the internet and AI, to their fullest potential. Neglecting to do so is 


akin to attempting to run in high heels. 


An Ideal State 
Let's consider the “ideal state” of humanity's relationship with AI. Recently, the Not Boring 
Substack”® remarked on the ideal state of a technology: 


Any technology that is sufficiently valuable in its ideal state will eventually reach that ideal 


state. 


By ideal state, I mean the imagined end goal or highest potential that technology could 


achieve if all the kinks were worked out and it was deployed ubiquitously. 


Understanding the ideal state is maybe the most important thing to do early in the life 
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of a technology, because if that ideal state represents enough value to enough people, the 


kinks will be worked out and the technology will be deployed ubiquitously. 


We must, however, ask: ideal state for whom? Some technologies may never reach their 
ideal state for humanity if it contradicts the interests of powerful market players. But if we 
can avoid market manipulations and ensure that the ideal state aligns with humanity's 
interests, it can prevail. 

In exploring the ideal state of the relationship between humanity and AI, we've 
uncovered a compelling narrative situated at the convergence of AI's immense potential, data 
sovereignty, and the realm of personalized AI systems—a narrative that absolutely warrants 
sharing. 

Visualize a future where each individual wields an exclusive, bespoke OS—an 
ingenious constitutional AI system meticulously honed to usher them through the 
transformative journey of Al-Assisted self-actualization. Envision crafting a constitution as 
distinctive as your very essence—a personalized compass empowered by AI's limitless 
capabilities, enriching every twist and turn in the tale of your life. This innovative concept of 
Al-Assisted Self-Actualization seamlessly marries advanced technology and the evolution of 
your personal trajectory—an illumination of AI's power to unlock latent talents and ignite the 
spark of loftiest ambitions. By embracing this forward-thinking ideology, individuals embark 
on a voyage of profound change, uncovering hidden potentials and transmuting aspirations 
into tangible reality. 

In an epoch marked by colossal shifts, this approach integrates AI into the very fabric 
of not only individual but also collective human progress. The profound ramifications of 
customized AI can also transcend personal spheres to encompass families, teams, 


communities, and societies, uniting them in a shared Al-assisted quest for advancement. 


How do we know it’s possible? 


Our experience using ChatGPT and other AI chatbots in the last 10 months has unearthed 
immense value and productivity gains despite only having access to data provided in the 
session. As mentioned earlier, my experience with AI has been transformative, enabling me to 
write faster and resume coding after a long hiatus. Imagine what a personal AI could do with 
access to all of one's data from online activity, messages and emails, purchases, voice 


conversations, physical locations, wearables, medical records, and more. 


Yuval Noah Harari, an Israeli author, public intellectual, historian, and professor, has 
written extensively on AI. He sees AI as an "alien species" that could trigger humanity's 


extinction. In a Wired interview2’ with Tristan Harris, Harari recounts a personal story about 


99 


his own journey of self-discovery, admitting that he was unaware that he was gay until he 


was in his twenties. 


“T realized that I was gay when I was 21, and ever since then I was haunted by this thought. 
What was I doing for the previous five or six years? I mean, how is it possible? I'm not 
talking about something small that you don't know about yourself—everybody has something 
you don't know about yourself. But how can you possibly not know this about yourself? And 
then the next thought is a computer and an algorithm could have told me that when I was 14 
so easily just by something as simple as following the focus of my eyes. Like, I don't know, I 
walk on the beach or even watch television, and there is—what was in the 1980s, Baywatch 
or something—and there is a guy in a swimsuit and there is a girl in a swimsuit and which 
way my eyes are going. It's as simple as that. And then I think, What would my life have been 
like, first, if I knew when I was 14?” 


Harari goes on to say that Facebook or Amazon can know that today's teenager is gay long 
before they do just based on analyzing patterns even with today's "stupid AI." He speaks to 


the advantage that such information can give a brand: 


“Coca-Cola knows this about me and shows me a commercial with a shirtless man; Pepsi 
doesn't know this about me because they are not using these sophisticated algorithms. They 
go with the normal commercials with the girl in the bikini. And naturally enough, I buy Coca- 
Cola, and I don't even know why. Next morning when I go to the supermarket I buy Coca- 
Cola, and I think this is my free choice. I chose Coke. But no, I was hacked. 

I often ponder how AI might ‘hack' for our benefit? Could our personal assistant help us 
uncover previously unknown patterns, make imperceptible correlations between our 
behaviors and outcomes, and build better mental models? With a better understanding of 
patterns, correlations, and mental models, we are more informed, and our decision-making 


could improve, potentially leading to significant value if not personal transformation. ” 
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In the near future, the idea of having one or more virtual coaches will not seem as odd as it 
does now. Imagine getting reminders, optimized for your engagement, that keep you on track 
to meet your goals. Or having candlestick charts for aspects of your health and relationships, 
with expert technical analysis of areas of support and resistance, outliers, and trend changes. 
Imagine having an ongoing heart-to-heart conversation with someone who exists solely to 
help you self-actualize and will not betray your trust. Or a personal learning assistant that 


develops and steps you through a personal learning plan for whatever you want to learn. 


a) How could you benefit from a personal AI that was dedicated to your personal betterment 


as you define and on your terms? 


Here are four examples of how it will help me: 


® Health Guidance: My personal AI knows that I don't like going to the hospital, and 
it's not something I typically do. So, it avoids recommending doctor visits as a first line of 
defense. However, when it deems it essential, like for prostate exams, colon health checks, 
and more, it encourages me, helps me schedule appointments, sends reminders, and even 


follows up after the visit to gather doctor's insights—all securely recorded in my vault. 


° ‘y Fitness Goals: I'm looking to get into better shape. Not that I'm in bad shape, but I 


want to be in great shape. My personal AI helps me set goals, tracks my progress, and 


provides reminders and notifications to keep me on the path to achieving my fitness goals. 


*  §) Data Management: I'm open to selling my anonymized data but only to reputable 
brokers. I also want to buy access to specific datasets that align with my goals or gain 
access to such data by participating in a community or data cooperative. My personal AI 
searches for opportunities and presents them to me in a way that makes it easy for me to 


understand and decide how to proceed. 


° wre Relationship Harmony: My personal AI contributes to healthy relationships by 
analyzing past experiences and flagging potential red flags in new relationships, 
encouraging caution or seeking advice when necessary. It assists me in defining and 
communicating personal boundaries clearly to ensure mutual respect and understanding. It 
also provides insights to avoid repeating unhealthy relationship patterns and make choices 
aligned with long-term well-being. My personal AI also Effective offers tips, techniques, 
and reminders for open, constructive communication, including drafting difficult 


conversations and handling sensitive topics with care. 
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As mentioned in the latest CQ substack: 


FOBO, a.k.a. 'Fear of Becoming Obsolete,’ is growing among college-educated workers. A 
new Gallup poll shows the percentage of college graduates who are worried about technology 
stealing their jobs has jumped from 8% in 2021 to 20%. Workers who make under $100,000 
also saw a significant 10-point increase to 27%. And young people between the ages of 18 
and 34 are also more anxious about it than older folks. As for those workers without a college 
degree, their level of concern about being replaced by technology has remained the same at 
24%. 

Consider the value of an AI that helps people whose jobs are made obsolete figure out 
their next steps, provides a path for learning what they need to know, and guides them on 


growing and positioning themselves for a successful transition. 


Outsourcing Awareness to Al 


The article Four Questions that Impact All of Us raises a thought-provoking inquiry: To what 
extent can algorithms reveal insights about us that currently elude our awareness? This 


question has captivated AI researchers for decades. 


ia) Thought Experiment: Is it possible that your entire data with a personal AI could help you 
create more value by helping you make better choices than the value that corporations 
generate by using your data to try to get you (or others) to buy their product? The good news 
is you can have both. 


i) Prediction: If we go this way, our personal data/AI will become the most important asset 
and tool in our lives. 

Even in our contemporary context, it's conceivable that algorithms could identify a wide 
range of conditions, from early-stage depression to cancer, thereby profoundly impacting our 


lives. 


Yet, beyond issues of physical and mental health, we should also consider the broader 
possibilities that extensive data analysis may unveil. Certain aspects of the human experience 
remain constant regardless of culture, generation, or social status. As our analytical 
techniques advance and our data resources expand, we might discover shared human 


experiences that connect us not only with our local peers but also with distant global 
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counterparts whose lives markedly differ from our own. 


However, the article highlights two significant challenges that persist in this brave new 
world. 

One concern is the act of bestowing authority upon algorithms, regardless of their 
intelligence, to provide insights about our lives. This can be especially problematic when 


algorithms discourage scrutiny and validation. 


As Hannah Fry, a distinguished mathematician at University College London and 
author of "Hello World," cautions, once something is cloaked in the guise of an algorithm or 
artificial intelligence, it can acquire an air of unquestionable authority. For instance, when an 
algorithm predicts a medical condition like cancer, we can seek confirmation through medical 
testing. However, when it comes to more intangible information, like our level of popularity, 
we may uncritically accept algorithmic assessments as truth due to our inability to 
independently verify them. This uncritical reliance on potentially flawed algorithms could 


lead to suboptimal decisions and outcomes. 


Moreover, there's a looming concern about algorithms possessing deep insights into our 
personal lives and sharing this information with various entities rather than with us, the 
individuals who generate the data. Platforms like Facebook, advertisers, and government 
agencies already have limited access to our data, and they may covet access to our personal 
AI and the comprehensive dataset it holds. This situation poses a significant risk, potentially 


compromising our privacy and autonomy. 


Personal Al and Data Sovereignty: A Symbiotic Relationship 


Personal AI cannot exist in a vacuum, divorced from the complex reality of the Attention 
Economy, where internet platforms, extractive algorithms, data brokers, advertisers, hackers, 
and scammers all play a role in shaping our online experiences. To truly harness the power of 
personal AI while safeguarding our privacy and autonomy, we must also embrace the concept 


of data sovereignty. 


In an era where data is akin to a valuable currency, the importance of data sovereignty cannot 


be overstated. It serves as the bedrock upon which the promise of personal AI is built. 


Contemporary discussions about data ownership, fueled by documentaries like "Social 
Dilemma" and "The Great Hack," as well as congressional hearings involving internet giants, 
primarily revolve around the central question of who controls and owns data. However, the 
battle for data sovereignty commenced long before it entered mainstream consciousness. 
Early advocates for privacy and encryption, such Eric Hughes who wrote the Cypherpunk 


Manifesto, were essentially advocating for data sovereignty—a principle that empowers 
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individuals with complete control over their digital assets, including their personal data. 


The foundation for achieving data sovereignty lies in cryptographic tools and 
blockchain technology. Picture a secure digital vault accessible through a decentralized 
identity. In this vision of data sovereignty, individuals are bestowed with complete 
ownership, oversight, curation, and dominion over their data. 

Similar to holding the keys to a physical vault, a decentralized identity grants 
individuals access to their digital repository, where their data resides under their personal 
jurisdiction. They possess the authority to explicitly grant or deny data access to their 
Personal AI and any other tools or systems of their choosing. If, for instance, the Personal AI 
seeks to share data with a third party, individuals must sign off with their private key to 
approve such data usage. With this protective layer of decentralized identity, the system 


cannot expose personal information or surrender data to external authorities. 


Introducing the Metaweb: a Safe Al-Assisted Environment 


Our envisioned future revolves around personalized AI systems thriving within the sphere of 
data sovereignty. This fusion holds the promise of transforming our lives, elevating decision- 
making capabilities, and granting us the tools to navigate the dynamic landscape of our 
world. However, for this vision to flourish, it necessitates an appropriate container. The 
current web infrastructure falls short of supporting this vision due to its compartmentalized 
nature, with applications isolated within their individual security, identity, and data realms. 
An overarching framework capable of accommodating the data access and protection 
requirements of the ideal personal AI state is the missing piece of the puzzle. This is where 
the Metaweb steps in. 


The Metaweb, a visionary concept outlined in the book "The Metaweb: The Next Level 
of the Internet," presents a safe, decentralized environment above the webpage known as the 
Overweb. By embracing the Metaweb, we unlock new possibilities for a democratic future, 
one where individuals have unprecedented control over their digital lives. 

Within the Overweb, individuals have their personal vaults, accessible only by them, 
their personal AI, and those they grant access to. Imagine having control over all your online 


activity data, even beyond the reach of platforms like Facebook. It's your data, on your terms. 


Accessible via a browser extension, a SDK or code library that loads with the webpage, 
and in the future browsers that adopt the protocol, the Overweb allows you to collect all your 
data across online activities, making it available for your best uses. 

This paradigm shift isn't just about the individual. It has the potential to reshape entire 


communities, as people share insights and support each other's growth. Additionally, 
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communities can have their own community AI that works with members and their personal 
Als to advance the community’s purpose. Imagine a world where AI-assisted self- 
actualization is the norm for individuals and communities, where everyone has access to the 


tools and resources needed to lead fulfilling lives and contribute to the collective well-being. 


But to fully realize this vision, we must address several crucial aspects: 


1. Data Ownership: Data is the lifeblood of AI, and in the Metaweb, data 
sovereignty is paramount. You should have complete control over your data, deciding who 
can access it and for what purposes. This ensures that your privacy and autonomy are 


preserved. 


2. Ethical AI: As AI becomes an integral part of our lives, it must adhere to strict 
ethical guidelines. AI should never manipulate or deceive users, and it should always act in 


the best interests of individuals and society as a whole. 


3. Transparency: Trust is essential in the Al-driven world. Algorithms and AI 
systems must be transparent in how they make decisions, and users should have the right to 
understand the reasoning behind AI-generated recommendations. This is best accomplished 


with open source software. 


4. Appropriate Constraints: In the Metaweb, AI must operate within well-defined 
constraints. A key component of this can be implemented through Constitutional AI, which 
establishes explicit guidelines and limitations for AI systems, ensuring their adherence to a 
framework of ethical and responsible principles. Given the distinct nature of the Metaweb as 
a web3 environment above the Web, it possesses the capability to enforce rules of conduct 
and engagement through smart contracts, effectively constraining AI behavior to align with 
permissible standards. 


5. Collaboration: Building the Metaweb and advancing AI should be a 
collaborative effort involving technologists, policymakers, ethicists, and the general public. 


Together, we can shape the future of AI in a way that benefits humanity as a whole. 


In conclusion, embracing the Metaweb and personalized AI systems offers a pathway to a 
future where individuals are empowered to reach their full potential, make informed 
decisions, and lead meaningful lives. It's a future where technology serves as a tool for self- 
actualization rather than a source of fear or control. To make this vision a reality, we must 
advocate for data sovereignty, ethical AI, transparency, appropriate constraints, and 
collaboration. By doing so, we can unlock the true potential of AI and create a more equitable 


and empowered world for all. 
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So, as we stand at this crossroads of possibility and responsibility, I invite you to join the 
movement towards the Metaweb and AI-assisted self-actualization. Together, we can shape a 
future where technology amplifies our humanity rather than diminishes it, where the power of 
Al is harnessed for the benefit of individuals and society. It's a journey that begins with 
embracing the potential of the Metaweb and personal AI, and it's a journey worth taking for 


the betterment of us all. 


New Terms: 


Ca AI-Assisted Self-Actualization: This approach leverages AI to help individuals achieve 


their full potential and realize their aspirations. It combines the concepts of AI assistance and 
personal growth, offering a clear and positive message about leveraging technology to 


enhance one's life journey. 


E Constitutional AI: Constitutional AI refers to AI systems with predefined guidelines and 


limitations, ensuring ethical and responsible operation. These systems can automatically 


reject outputs that violate specific ethical or legal standards, guided by heuristic imperatives. 


@® Data Sovereignty: Data sovereignty empowers individuals with complete control over 
their digital assets, including personal data. With data sovereignty, individuals have full 


ownership, oversight, curation, and dominion over their data, ensuring privacy and control. 


fae Highest Use Theory: According to this theory, as technology evolves, humanity must use 


its tools at their highest potential to fulfill their promise and honor the gift of advanced tools. 


This principle applies to AI and the Internet, emphasizing maximizing their positive impact. 


LL) Life Literacy: Life literacy encompasses becoming literate in crucial aspects of life, with 


or without AI assistance. It includes various literacies such as food, health, communication, 
media, attention, data, and conflict resolution, enabling well-rounded understanding and 


competence in life. 


Ei The Metaweb: The Metaweb represents the space above the webpage—a meta-layer 


over the Web enabling interactions, computations, and experiences accessible through a 
browser overlay. It extends to virtual worlds, the physical world, and the conceptual world of 


words and ideas. 


fase The Overweb: The Overweb is a decentralized public space above the webpage, serving 


as the first full instantiation of the Metaweb. It's a hyper-dimensional web layer where people 


can meet and interact directly and through content, ensuring safety and accessibility. 
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ee Personal AI: Personal AI is a digital twin for humans, dedicated exclusively to 
individuals' goals and needs. It operates in a private, personal space, assisting in making 
choices, pursuing goals, and monitoring progress in unique and effective ways. 

& Personal Vault: The personal vault is a secure storage for all of one's data, protected by 


blockchain or similar technology. It continuously accepts incoming data streams, ensuring 


data ownership and protection. 
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Al and the Editor 


Sophie Whittle, James O'Sullivan, Michael Pidd 


Digital scholarly editing remains an industrial craft: the materials, medium and methods are 
technological, but the work itself remains largely manual and bespoke. And because digital 
editions are labour intensive, they can be limited in scale. Editors—that is, textual scholars 
and the makers of editions—were among the first in the arts and humanities to recognise the 
publishing affordances of the digital. And so it is surprising that machine learning and natural 
language processing have not yet played a greater role in scholarly editing; that newer forms 
of computation have not advanced editions to the same degree as markup languages did in the 


final decades of the twentieth century. 


It can be easy to get caught up in looking only at the latest, most novel forms of 
artificial intelligence, for example, using OpenAI’s famed ChatGPT to assist with annotations 
and contextualising materials. But even long-established symbolic and statistical natural 
language processing techniques could be used to semi-automatically classify and understand 
the context of words and terms, allowing other kinds of insights into sources. Text collation 
to identify variants has benefited from computer assistance [1], and certainly, while methods 
which offer fresh (ie. data-driven) insights are naturally privileged over those which can only 
offer expediency, there is substantial value in computational tools that can reduce the amount 
of routine—even tedious—tasks that are required by digital scholarly editors. Natural 
language processing can assist with time-consuming editorial tasks such as annotating, 
glossing and connecting texts; such a return not only shortens timeframes to publication (and 
thus, public consumption), but also allows more time for critical examination (the bit that 
makes the editing, scholarly). One can only assume that the general lack of adoption of such 
tools and techniques among scholarly editors is a consequence of ideological opposition, lack 
of access to the necessary computational expertise, or simply ignorance of their value; 


certainly, the opportunities are there. 


There are many reasons why one would object to the use of AI in editing: the (often 
valid) concerns over representativeness and accuracy of semi-automated processes [2], and 
the lack of transparency in the production of (some) tools and processes themselves. The 
introduction of AI to scholarly editing processes might involve additional work on the part of 
the editor to ascertain the sources and methods adopted by the model. Editors should also be 
mindful that any rush to AI might replicate the consequences of the rush to OCR and mass 
digitisation that occurred in the 2000s and 2010s, which precipitated poor quality outputs of 


limited use for research, squandering vast amounts of resources and labour. 


108 


Yet, there is still value in collaborating with AI, despite having to maintain some of the 
necessary work of critiquing the output. While there are shortcomings of manual validation in 
terms of slowing down the technological task in the short term, the method of curating 
detailed prompts to input into AI models—and subsequently investigating what the AI has 
come up with—can in fact enrich digital scholarly editing and make the process quicker 


overall. 


At the most basic level, one can certainly see potential in AI as a means of enabling 
editors and their audiences to intuitively approach their curated materials through distant 
ways of reading [3]. The majority of text-centred activities in the digital humanities fall into 
three broad categories: the assemblage of textual resources (including digital scholarly 
editing), the sharing of such resources (digital publishing), and the use of computer-assisted 
techniques to quantitatively analyse cultural materials (distant reading or cultural analytics). 
Katherine Bode criticises the digital humanities for having separate “curatorial and 
statistical” dimensions, a disciplinary culture which is split between those who gather and 
edit, and those who analyse with machines [4]. That the aforementioned practices continue to 
develop in isolation has only served to slow the progress of each: digital scholarly editions 
and the practice of digital scholarly editing would benefit greatly from integration with data 
mining techniques and machine-assisted insights, and methods for computer-assisted analysis 
are only as good as the data on which they operate. 

Furthermore, in a survey designed to better understand the expectations and use of 
digital editions, participants were asked, “what use would you make of the data published in a 
digital edition”, to which the most frequently cited response was “teaching”, and “text 
analysis” as a very close second [5]. Text analysis, like the edition, is changing, and as text 
analysis of the quantitative sort becomes increasingly prevalent, so too will the demand for its 
integration with digital editions as digital systems (which they are). In some instances, users 
are permitted to download the materials from editions and can subsequently conduct their 
own analyses if they have the specialist expertise or resources required to do so; but in most 
cases, it is either impossible or prohibitively cumbersome to use the data contained in digital 
editions for computer-assisted text analysis. 

The future of digital editing should be one in which the publishing platforms are 
integrated with statistical methods already being used in the digital humanities. Such a future 
would move scholarly editing, and indeed, the wider digital humanities, beyond the two 
dimensions identified by Bode. Amy Earhart believes such a future is possible: “data sets and 
editions can coexist, but only if those from digital and textual editors can find bridges to 
those approaching digital humanities from other traditions and with other goals” [6]. Seeing 


the creation of a digital scholarly edition and the application of digital methods for content 
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analysis as part of a holistic approach to knowledge dissemination would not diminish the 
intimacy of the editing process, but rather, supplement it. The combination of AI alongside 
traditional curatorial methods provides editors and audiences with different perspectives, 
specifically, the type of quantitative evidence that, for better or worse, is valued in today’s 
society as either a form of evidence or a point of entry into complex information. Embedding 
Al-driven cultural analytics in editions themselves democratises distant reading, as those 
wishing to apply such methods to the contents of an edition would be able to do so without 
the need to develop or access specialist expertise or software. And it brings reliability and 
credibility to datasets. One of the great challenges of distant reading is that methodologies are 
only as reliable as the data being tested, and in scholarly editions, we find ideal datasets 
which have been expertly, and more importantly, transparently (in that the profile of their 
curator is visible and human), compiled. 

If the ambition of digital scholarly editions is to make digitised text more accessible 
and searchable, it seems that a PDF of a printed text, archived and well described in a suitable 
repository, would be sufficient. But if the ambition is to use the digital to transform scholarly 
editing to a more radical degree, then it would seem that the ways in which critical editions 
can be read is an obvious opportunity, particularly as scholars across the digital humanities 


and AI have already developed, adopted, and tested a range of methods for doing just that. 


For editors who have a preference for textual editing traditions, their editions can still 


exist as print or as digitised editions. 
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Phantasmagorical simulation theory of Metaversal 


Literature 


Yohanna Joseph Waliya 


The advent of digital hypertext, internet and metaverses has made the 21st century e-lit 
readers swinging in the midst of four worlds of interactive apprehension. The four worlds are 
viz: the real world where human and hardware interact physically with each other in the 
software interface using synthesisers such as mouse, keyboard, screen, VR controllers; the 
fictional world, a spatiotemporal imaginary realm created by inner trajectories of thoughts 
and visuals as a result of appreciating work of arts like poetry and novel ; the virtual world or 
digital world which is birthed by digitality leveraging computer networks[1]; and lastly, the 
extended reality world where both the readers and the text are datafied as virtual 
embodiments engaging each other[2]. This concept establishes the metaverse as a mixture of 
real world; virtual reality world, augmented reality world and artificial intelligence developed 
via technologies [1][3]. In this context, the materiality of the text and its readers are totally 
integrated. This phenomenon of blurring the multimodalities defines phantasmagorical art. 
Thus, explains the imagination of the technological singularity for both human being and text 
on the cyberspace. Singularity is the major creative characteristic of the metaversal literature, 
and its literary utopia which projects how-the-world-should-be of the four worlds, it 
represents. Metaversal literature is equally characterised as hapticoverbivocovisual i.e., the 
contents of the literature are composite of haptics, words, audio and videos. 

Metaversal literature is an immersive ludokinetic oeuvre experienced via digital self- 
imposed hallucination on the generated extended reality (XR) environment using animated 
avatars with the help of computer rendering technologies. Abramson conceived the idea of 
metaversal literature in 2015 to mean that it is a collaborative space of fusing binary 
opposites such as truth and falsehood, sincerity and irony etc. Its storyworld encompasses a 
fusion of original, borrowed, and reconfigured content, alongside genuine, hired, and 
reimagined perspectives. This storyworld aims to create heartfelt, self-revealing literary 
works that serve both as a means of personal introspection and as a significant expression of 
political engagement. [4]. On the other hand, what concerns Song, it is the fact that 
metaversal literature is a user generated content community of creators, users themselves and 
work-world on the metaverse whereby authors immediately experiences all literary roles and 
situations with the technology sharing avatar to truly attain empathy between themes in order 
to accept the mission of preventing technology risks and defending the goodness of human 


nature [5]. Unfortunately for Song, she based her finding on the traditional theories of 
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literature not electronic literature. The characteristics of virtual reality, she stated such as 
immersion, interactivity and imagination are the same characteristics of electronic literature 
which metaversal literature is its subset. 

Dreyfus called such phantasmagorical art an animated hallucination i.e. accurate 
embodied synaesthesia or an exciting revelation of natural wonders suddenly appearing to the 
observer because of some clarification in his or her attention or some shift in the atmosphere 
or angle of vision by cameras, cinema and other mechanical illusion [6]. The concept of 
Dreyfus’ perception of phantasmagoria in 18th century is similar to the Gesamtkunstwerk 
which Ayiter sees it as Romantic period concept referring to synesthetic union of all arts-from 
music, dance, and poetry, to visual arts and stagecraft. Ayiter continues to invoke David 
Morgan’s four basic aspects of a Gesamtkunstwerk such as variety of media, unity of effect, 
physical environment and viewers’ participation [7]. These Morgan’s basic aspect of 
Gesamtkunstwerk identifies the characteristic of the metaversal literary storyworld whereby 
the metaversal reader (i.e. Mreader henceforth) participates in synesthetic embodiment in the 
storyline by controlling the narratives to suit his desire, acting as structurer of the arch of the 
stories. 

Inversely, it is evident that people owning the metaverses could use the algorithmic 
influences to control, map the geographical narrative and the thought patterns of the users to 
achieve their socio-political ideologies via metaversal literature just like the traditional 
writers who have been shaping our cultures from times immemorial turning their utopia into 
reality as a means to subtly force the society into their desired ends. Moreover, humanity is 
the only constant element in all stories even with the interception of digitality, internet and 


metaverse that rendered natural human posthuman. 


The posthuman being is fully manifested in reading metaversal literature. The literary 
Mreader extends from his natural tripartite to quadripartite whenever he launches himself into 
the metaverses to read. His avatar connects with his soul, spirit and body to make him a 
complete quadripartite posthuman. To this effect, his body connects with the real world, his 
soul links to the fictional world, his spirit imbeds in the virtual world whilst his avatar 
projects into the augmented world simultaneously when he performs Mreading. Therefore, 
Mreading erases space and time by imposing on the Mreader such a cosmological projection 
via the three-dimensional virtual reality headsets, glasses and haptic controllers. These invoke 
conscious oneiric world experience. One regains oneself when the XR glasses and headset 
are removed or disconnect from the metaverse. 

In this regard, human loses its post-humanity when it stops interacting with metaversal 
literature. It is evident that metaverses are birthed from astral soul projection experience via 


rendering technologies. It is the reason avatars know one another, interact with each other and 
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meet with each other as humans which they represent on the physical world. These 
phenomena materialise the collusion of the Plato’s world of forms with the physical world via 
metaverses 

Therefore, the metaversal literature is the evolving future text that rooted in modelling 
the four worlds as mentioned above alongside the creator’s utopia, and the participation of 
the posthuman on the informatic generated ecosystem. Metaversal literature shifts a bit from 
the general 2D electronic literatures on the dynamics Web 1.0 and Web 2.0 for it is built on 
the Web 3.0 making its stories felt, touched and sighted altogether via virtual reality 
synthesisers. Thus, phantasmagorical simulation theory of metaversal literature is explored in 


the metaversal literature definition and its storyworld. 
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Open Source and Non-linearity - A futuristic 
Virtual Reality way of dealing with News in “News 


In Time And Space” 


Julian Gagel, Timo Liittig, Theresa Berg, Jasper Hustedt, Giuseppe Abrami, Alexander 
Mehler 


1 Introduction 


Information has always been a driving factor in society, today more than ever. Enabling 
people to share information and stay informed allowed society to evolve beyond its 
limitations at the time. Revolutionary technologies shaped the way we as individuals interact 
with news and information, from the printing press creating newspapers, radio transmission 
enabling audio and video broadcasts, to the internet connecting devices and people all around 
the world. 

News agencies are the main source of information for most individuals, but the 
limitations of each medium through the ages meant they were always facing challenges. 
Newspapers are rather limited in capacity so the local news outlet creating the paper had to 
decide what information is relevant and worth putting into print. A similar challenge persisted 
with radio and TV news broadcasts since airtime is limited and needs to be planned and 
produced. The digitalization of information pushed news publishers to provide their reports 
online via websites or apps which lead to the creation of news feeds that take reports from 
different sources and topics and present them to a user. But the problem of relevance is one 
that’s greater than ever. In such a fast moving medium the focus on always having the 
newest, most interesting, most attention grabbing information grew and grew. So the problem 
of judging what information is deemed important enough to present to individuals is still 
persisting but it is now being evaluated by algorithms that aim to personalize these feeds to 
certain user groups. This means that any individual is dependent on another entity, like an 
algorithm or directive board, that decides what is deemed relevant and worth presenting. 
Even with the capabilities of today’s technology this massively limits the number of available 
sources for news reports a person is presented with. News feeds for people in western 


countries for example mostly present news from western news outlets. 


A big question arising from this is whether there still is a need for a separate entity to 
decide what is relevant enough to be brought to attention? With such advanced and fast 


access to information the possibility of bringing the power of deciding what is important back 


116 


to an individual is achievable. What is needed now are the tools to enable everyone, 
companies and individuals alike, to allow them to explore this data for themselves rather than 


relying on somebody else to decide what is and isn’t relevant. 


2 GDELT as our source 
We have chosen GDELT[1], Global Database of Events, Language, and Tone, as our main 


source for our project. GDELT is a large database that collect a wide range of media content 
and especially news sources. The complete dataset is enhanced with additional information 
like sentiment analyses and extracted themes. One of GDELTs greatest strength, the 
unfiltered access to categorized events from all over the world and different outlets, is also 
one of the main weak points. As the user is provided with mostly raw data it becomes hard to 


evaluate the quality and relevancy of each datapoint. 


3 News In Time And Space 


To propose a way of unbiased and nonlinear user interaction, we developed our program 
“News In Time And Space” [3], in short “NITS”. NITS is a virtual reality tool which is used 
to present data from the GDELT project in a highly interactive manner. Interacting with large, 
unfiltered data can be confusing and even frustrating which makes it inaccessible to most. 
Since virtual reality became more widespread, its main focus has been the ease of and 
building upon user to system interaction (e.g. [1, 4]). Systems to make interaction with large 
amounts of data in virtual reality possible aren’t fully researched, but appealing to our cause. 
Users can have a more hands-on approach, which is more stimulating than regular tools and 
might spark interest to further their exploration. Virtual reality presents a more intuitive way 
of human-computer interaction by immersing the user in a virtual environment and relying on 


natural gestures (Figure 4). 
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Figure 5 Info of an Event 


To make large amounts of data approachable via NITS one needs to be able to specify what 
they want to see and interact with. For this, filters have been implemented on the basis of the 
virtual reality environment. In order to specify relevant events there is a set of filters 
available like a timeline to specify when events occurred, the types of events to look at and 
participating actors like countries, people or institutions (Figure 2). Other metrics like the 
tone of news reports or their value on the Goldstein scale, an estimation of the impact an 
event had on the stability of a country, allow a user to compare differences in news coverage 
from different origins (Figure 1). Users can easily mix and match these filters interactively, 
allowing for seamless exploration of the available data. Events are placed on a globe at the 
location they took place at and are connected to the location of the participating actors 
(Figure 3, 6 ) This means that events present a small network of linked locations that imply 


their global or regional impact (Figure 6, 5). These tools try to enable a user to explore, adjust 
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and interact with the available information by themselves in an unfiltered and not globalized 


way and make big data accessible to everyone. 


4 The future of text is open source 


In these days of information overload filtering relevant information is necessary. But who 
determines whether information is relevant? In our opinion: The user. A powerful search 
engine for information can never tackle the lack of transparency if its generated output of the 
underlying algorithm remains closed to the public. To understand how information is filtered 
it is necessary to get a view on the code base determining the importance of information. This 
should be the basis of information freedom in a digital world. Therefore, we advocate open- 
source solutions in this field and our project and code is open to the public on GitHub[2]. 
However, our data relies on GDELT. Though GDELTs files are available to everyone, some 
of the underlying codebase of algorithms that filter information from news articles are not 
fully disclosed. As a consequence, we can publish our application open-source but we cannot 
guarantee completeness and accuracy of the database we use. Only fully open-source 
solutions create transparency and are thus an essential element for guaranteeing traceable 
results in the journalist and scientific sector. 

With our plea for a certain degree of user-centricity, we indirectly tie in with a classic 
hypermedia concept, namely that of adaptive hypermedia [2]. Specifically, we envision our 
system to be open in several ways: with respect to the underlying information source, e.g., 
online social networks, instant messaging systems, blog hosting services, news aggregation 
or discussion sites, or any other data repository that is at least temporally ordered and 
contains spatial information that can be localized in the way we have done with GDELT. In 
this way, the user is empowered to select the information sources that are of interest to her or 
him and that are to be made interactively accessible in VR by means of NITS. This requires 
the generic further development of NITS as a universal system for VR-based mediation of 
textual information with substantial temporal and spatial structuring. On the other hand, user 
adaptivity also refers to what is known from adaptive hypermedia [2], according to which the 
system builds a model of the intentions, goals, interests, and interaction history with the 
respective user, in order to offer information with different granularity and contextual fit, so 
that the interaction with the user becomes more central in the sense of human 
communication. With the advent of generative AI methods, we now see the possibilities of 
such adaptive systems, as the content of various media can be made subject to semantic 
modeling, as can the interactions with the user, in a way that contextualizes them reciprocally. 
In other words, user actions become an evaluable part of the interpretive context of 


information and vice versa, promising better, more user-adaptive information selection and 
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more natural interaction. At the same time, however, the requirements for transparency and 
openness of system decisions and their reconstructability by the user are increasing. This 
applies not only to the actual information sources and the developed programs, but also to the 
user model that is created in the course of using the respective VR system and its 
accessibility, controllability and modifiability by the user at any time. This conclusion leads 
us back to the term Open Source, but in a way that distinguishes between three aspects. In our 
view, the future of text means increasing user adaptability, which in turn brings user-specific 
processes of text comprehension into focus, and ultimately underscores the need for open 


source. 
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The “VutuRe of Text” is Virtual 


Shaduan Babbili, Kevin Bonisch, Yannick Heinrich, Philipp Stephan, Giuseppe Abrami, 
Alexander Mehler 


1 Introduction 


In the digital age, the way we communicate and interact with information has undergone a 
profound transformation. The ubiquitous use of text-based communication, from emails and 
instant messaging to social media platforms, has become an integral part of our daily lives. 
However, as technology continues to evolve at an unprecedented pace, the future of text is 
poised to undergo a revolutionary shift, one in which virtual reality (VR) stands as a pivotal 
player. 

As we embark on this exploration, it becomes increasingly clear that the convergence of 
text and virtual reality holds the potential to transcend the limitations of traditional text-based 
communication. The fusion of these two domains promises to enhance our ability to convey 
complex information, foster richer and more meaningful interactions, and revolutionize the 


way we consume and create textual content (see Figure 1). 


Welcome to the VikieibraRy! 


Viki LibraRy, is a first implementation for Ganesan 
exploring online informationbased on / Regie . 


Figure 1: Reading the book “Introduction” in the Viki LibraRy/ 
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2 Hypertext 


Hypertext is a foundational element in the evolution of textual information, fundamentally 
altering how we interact with and comprehend content. It refers to text displayed on 
computers or other electronic devices with references (hyperlinks) to other text, allowing 
readers immediate access to additional information. It is pivotal in enhancing accessibility 
and interactivity of information, providing dynamic linking to diverse and multidimensional 
content. This enables users to navigate between related topics, sources, and multimedia 
components seamlessly, offering a richer and more engaging learning and reading experience. 
Beyond its current applications, the integration of hypertext with augmented and virtual 
reality technologies is set to redefine the manner in which we perceive and interact with 
textual information. In such environments, hypertext can create immersive, three-dimensional 
information spaces, allowing users to explore and interact with information in novel ways 
(e.g. [4, 1, 5]). This integration has the potential to spawn groundbreaking applications in 
education, entertainment, and various professional fields, offering more intuitive and 


engaging methods to visualize and understand complex information. 


3 Viki LibraRy 


To contribute towards a future of text where virtual reality plays a pivotal role, we present the 
“Viki LibraRy: A virtual reality library for collaborative browsing and navigation through 
hypertext” [2]. As for our first implementation, we used Wikipedia as a hypertext system. The 
primary idea of our first implementation of Viki LibraRy is to visualize Wikipedia in a 
dynamically built and fully immersive VR library. To do so, we project Wikipedia’s 


categories as rooms and articles as books within virtual reality (see Figure 2). 


123 


Figure 2: A glimpse into one of Viki LibraRy s rooms. 


The user can stroll through rooms, picking up books from the shelves and reading them all in 
virtual reality. In order to traverse multiple rooms and reach information of different 
categories, portals are placed within the virtual world to connect the different layers and 
rooms (see Figure 3). By visualizing the information provided through Wikipedia in a more 
immersive and engaging environment, we hope to encourage users to do more than just skim 
or select articles, but to have the experience of strolling through a beautiful library and 
randomly picking up books that catch their eye. 

But we also acknowledge the need for a systemically structured search function, so we 
incorporated a search desk into the virtual library for quick research. We also provide a 
minimap for the user to better understand the library’s structure and comprehend the rooms’ 
layout more quickly. 

As we already alluded to, the Viki LibraRy is a dynamically built library in the sense 
that each room is built independently and corresponds to the category it is representing. An 
algorithm forms each section of the library based on the current information context. By 
doing so we ensure that rooms have the correct size and structure for the information they are 
visualizing, and it further allows the usage of different hypertext systems than Wikipedia. 
Hence, the Viki LibraRy allows the incorporation of various hypertext systems such as social 
media platforms or chat-systems. 

Since text-based information is more fun when shared and experienced collaboratively, 
we built the Viki LibraRy to support multiplayer, by using Ubig [3], which means that users 


can invite each other into their library to traverse through it together. 


124 


People 


Dhilosophy 


Figure 3: Looking at the two portals that lead to the categories “People” and “Philosophy”. 


4 Conclusion 


We presented the Viki LibraRy, a virtual reality library designed for immersive traversal 
through hypertext. Our exploration illuminates the transformative potential of integrating text 
into virtual reality, which will undoubtedly play a significant role in shaping the future of 
text. The Viki LibraRy exemplifies how the fusion of text and virtual reality can enhance 
information access, foster interactive learning experiences, and redefine collaborative 
knowledge sharing. It serves as a testament to the ongoing evolution of human-computer 
interaction and the innovative possibilities it presents. 

As we look forward, the integration of text into virtual reality promises to revolutionize 
education, communication, and information dissemination. The Viki LibraRy provides a 
glimpse into this future, encouraging us to contemplate the profound impact this convergence 


will have on our society and how we engage with textual content. 
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Prolog: Foresight & XR — Imagining New Futures 


Strategic Foresight vs. Forecasting 

We often hear the term Foresight, or in a business context the phrase Strategic Foresight (to 
emphasize its business planning applications), bandied about when talking about planning for 
the future and they are often conflated with Forecasting which is the identification and linear 
projection of current trend lines. But the distinction between them could not be more stark. 
Forecasting is all about guessing what will probably happen in the absence of disruptive 
change, while Foresight is focused more on everything else that might happen in the face of 
such change! 


Foresight & Futures Studies as a Discipline 


Foresight is the application of the discipline of Futures Studies to surface & challenge current 
assumptions that limit our thinking so we can imagine a broad range of Possible Futures as a 
guide to how we act in the present to dream up and bring about preferred futures that we 
would actually look forward to living in quite unlike the dystopian mania that has come to 


dominate much of popular culture. 


The discipline of Futures Studies itself is the identification and study of : 


¢ Factors that shape what might happen in the future. 

¢ The Ranges of values those factors might take on. 

¢ The Degree to which they are independent of or constrained by each other. 
¢ The Action that could be taken in the present or future to influence them. 


¢ The Likely Consequences of such interventions and how other actors would respond to 


those consequences. 
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Possible Worlds 
Each Measurable Facet of the Set of All Possible Futures (including the history of events that 


led up to them) can be thought of as an independent dimension in a high dimensional Design 
Space. 

Each Point in that Design Space is a unique Future / Possible World — Think of a Sci- 
Fi Multiverse Concept where countless Worlds are indistinguishable from each because most 
facets and value graduations are inconsequential (e.g., whether you had coffee or tea at 
brunch last Wednesday and what temperature it was when you took your first sip). 

While most of these facets and the values thereof don’t matter in consequential ways, 
permutations of Consequential Facets can be Partitioned into named regions that focus on just 
a few factors at a time merging most specific values. The permutations of possibilities are 
nearly infinite and Expand Over Time as chains of events diverge to form a Futures Cone 


[34] (inspired by the Light Cone concept from physics). 


This observation lets us focus on just a few factors at a time and partition their possible 


values into nameable ranges — each of which encompasses a multitude of possibilities. 


Then we can Name these regions / partitions / sub-spaces or whatever one chooses to call 
them and juxtapose them as oppositional alternatives like “High Interest Rates” vs. “Low 
Interest Rates”, “High Unemployment” vs. “A Labor Shortage’, or “XR Use Dominated By 
Entertainment Applications” vs. “XR Use Dominated By Scholarly & Business 
Applications”. This lets us set up grids comparing a few factors so we can pick a possible 
world as an exemplar in each sector for comparison. 

There are a large number of individual techniques and methodologies for exploring 
futures like Frank W. Spencer IV and Yvette Montero Salvatico’s Natural Foresight® 
Framework [35] and Maree Conway’s writings at Foresight in the Present [36] and countless 
others that are but a search engine query away. 

What the best of them share is the goal of empowering one to break away from one’s 
assumption, so in that spirit, we can now consider just one of a multiplicity of possible 


worlds which might prove worth working toward. 
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A Scenario 


The Unboxing 

The year is 2048 I'm sitting in my office surrounded by bookcases, filing cabinets, and other 
paraphernalia of academia. In front of me is a newly arrived package from Apple containing 
the Apple?8 Vision Pro IIcx. I eagerly tear off the shrink wrap and open the box to reveal 
Apple's latest headset which unlike its predecessors doesn't require any form of tether and is 
scarcely any heavier than my regular eyeglasses which I remove and replace with the new 


unit. Touching the fob brings it to life. 


Vision Correction 

It doesn’t require any corrective lens inserts since it is able to map the geometry of my cornea 
to actively correct for my nearsightedness and astigmatism and, by employing a fine grained 
access color correction factor medical preference, compensate for my unique red-green color 
perception so it can boost the appropriate colors to let me see how things would look with 


normal color vision. 


The Light Shroud 
A fully-embodied Siri appears in front of me like a force ghost from Star Wars to provide 
initial start up instructions directing me to activate my Apple Reality Printer II which begins 
to hum softly as its lasers start to paint out layer after layer of a 3-D light shroud whose exact 
dimensions and geometry were registered at Apple when I was fitted for a unit at the Apple 
Store and transmitted to the printer when the unit’s delivery was logged by Planet Express. 
A few minutes later the shroud is ready; I take it out of the printer and it is only a few 
dozen millimeters wide at its thickest, but when it snaps into the frame of the headset it 
provides an absolutely perfect fit to my face; it is just right to make sure that no extraneous 
light sources are able to leak in at the edges of the headset improving its ability to project 
clean imagines into my peripheral vision. This minor bit of customized kit replaces the 
thicker rubber and foam light guards that weighed down early VR rigs and looks rather cool 


when mated with the burnt blue2? grade 5 titanium finish of the unit. 
g 


Vision Augmentation and Novelty Headset Customization 
The device’s countless sensors can provide the same full range of augmented spectrum 


overlay as a Predator?9 Helmet [37] but in such a sleek form that Apple fan boys have 
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nicknamed it a Geordi [38] much to Apple’s chagrin. Ironically, a “La Forge” app, which 
hacks the outward facing to display to replace the user’s eyes (which have long since been 
upgraded to Full Outward Pass Through to avoid the Uncanny Valley [39] effect that some 
users perceived in the synthetic avatar eyes of the original model) with a depiction of the 


banded ribs of the Visor worn by the popular Star Trek?! character. 


Headset Cafés, Mnemonic Couriers, & Personal Reconnaissance 
Drones 


I prefer to do most of my work from home, but if I had to go into the city, I would leave my 
visor home and opt to rent one for a session in a Headset Café — high end XR equipment is 
just too lucrative even as cannibalized parts to go wearing it on the streets or in a setting 
without adequate security. 

With the high levels of street crime and never ending reports of cyber security breaches 
in the 2020’s “Mnemonic Couriers” actually became a thing, but unlike William Gibson’s 
depiction of dependence on Neural Implants, they would rely on old fashioned human 
memory and stealth to get the job done, working in pairs traveling on different days and 
routes with each committing half of a long truly random one-time access key to memory. 
They wouldn’t wear flashy suits and could be any age, race, or sex and would look totally 
unremarkable. Critical data files would then be encrypted and XORed with the mnemonic 
key making it safe to transit them over insecure channels once the intended recipient 
confirmed that both couriers had already arrived safely and that the key halves produced the 
correct check sum when combined. To further mitigate the possibility of both couriers being 
subjected to duress, their progress would be monitored by a series of PRD’s — Personal 
Reconnaissance Drones. 

Sadly, when the first high end headsets came out and people started using them in old 
style coffee shops there were a number of ugly assaults with a similar MO. One or more 
assailants would brazenly stride up behind the victim, cosh the poor soul over the head, and 
make off with the headset — not surprisingly none of the onlookers or staff would risk a 


wack from a truncheon to intervene. 
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Multi Pass, Leasing High End Hardware, & Closing the Digital Divide 


The Headset Café put a quick end to that sort of thing. One could register for a block of time 
a few hours early and enter through a “Man Trap”. This was little more than a chamber with a 
pocket entrance door on one side and a similar exit door opposite to it. You would enter from 
the street, be sealed in and then insert your MULTI PASS ID card into a reader for 
verification speaking aloud a passphrase like “My voice is my passport; verify me.” Once 
voice and facial rec confirmed that you matched the records on file at MultiPass Inc. (which 
emerged to be the 3rd Party identity provider of choices for such things) the exit door would 
slide open and a voice would direct you to a VR Pod. Otherwise neither door would open and 
you could wait for the police to collect you for trying to use a stollen MultiPass, although in 
practice this would never happen. 

Thus the advantages of leasing time at a café were the security of knowing you could 
enter full emersion in total safety and that you could take advantage of more expensive 
hardware (like maxed out versions of the latest Apple or Meta or Open Hardware headsets, 
higher end omni-directional treadmills, and haptic suits) than lower income earners could 
afford to own. This made them an indispensable enabler of social mobility with government 
and private foundation grants supporting their deployment and use in some of the “worst” 
urban neighborhoods to head off the development of an XR “digital divide”. 


Spatial Keys, Redact Blue, & AppleStride 


Turning to my desk, I see a newly arrived package from my long term friend and colleague, 
Frode. It contains an intriguing 3-D printed copy of his prized shard of obsidian, so I issue the 
voice command “Recognize this!” as I slowly turn it over and rotate it in my hands to 
examine all of its surfaces. A soft “pon” signifies that the object’s geometry has been 
registered and Siri’s voice coos “Keyed Geometry, Payload Present’. Interestingly, aspects of 
the shape of the shard serve as BOTH an index bound to a URI that holds an encrypted block 
of transferable data and as a decryption key unlocking that data as soon as it reaches my local 
headset. 

I immediately utter, “Display Manifest” summoning up a static model of a 3 faced open 
cube representing Frode’s work office. The model room appears slightly dimmed and greyed 
out as if filled with a morning fog to indicate that the view isn’t “Live”. There are a number 
of papers on the desk that look like sheets of navy blue paper — known formally as 


“Redaction Blue” or just “Redact Blue” for short indicating that their actual content has be 
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flagged by a “Redact This” directive; had he further authorized me to be able to see them, 
they would have appeared with a Redact Blue edging. Similarly, several of the books on the 
wall behind his desk have Redact Blue spines. Unbeknownst to me, his “actual” desk 
contains several framed photos and art pieces whose Digital Twins have been excluded from 
the AR model and made invisible by his issuing a “Vanish This” directive after picking them 
up and rotating them until the system both registered their shape and appearance and captured 
a scan of the underlying surface on which they were placed. 

“Request Entry’, I say. Siri directs me to “Prepare for Full Immersion” which means 
that I have twenty seconds to get situated on my AppleStride Omni-Directional Treadmill 
which is the most expensive bit of kit I have at my disposal. Once I am in position there is a 


pronounced “Ping” and the voice coos “Initializing Transfer”. 


Access Control and Phasing 
Since I am already on Frode’s Access Control List (ACL) for his work office sim, my system 
has opened a persistent connection to his RealityServer allowing me to access high fidelity 
virtual copies of its contents as my office crossfades to the viewpoint I would experience if I 
was standing in front of his desk in the real world. 

Some items in the room have been marked as static “Scenery” and cannot be moved, 
but others are reactive elements that I can pick up, like the seating chart on an easel next to 


the window. 


I walk over to it as my treadmill keeps my real body centered near where I started, and I 
pick up the seating chart. Of course its stays put in Forde’s real office, but to anyone else 
entering the sim “in phase” with me, it would now be represented as a semi-opaque 
“Ghosted” residual image along with an avatar depicting me holding its virtual twin. 

The Phase concept is a way to permit multiple visitors to share the space without being 
aware of each other’s presence other than through the side effect of objects “Phasing Out” if 
someone else picks them up AND they are marked for “Solo Access”; objects marked for 
“Shared Access” would have “Tab Well” semantics and appear in their original location to 


new arrivals not in phase with other visitors. 
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Virtual Object Replication, The “Pen”, & New Millennium Open Metaverse Interop 
Protocol 

Mark Anderson connects a few minutes later and because we are lab mates we are rendered 
in phase with each other. Since we were planning to discuss the seating together, I employ a 
“Pulling Apart” gesture which tears the chart in half as a shower of sparkles cascade out of 
the edges of each half as each half re-forms into a perfect synchronized copy. 

I hold my hand up and say “Pen” causing a virtual stylus to appear in my hand which is 
ensconced in a Meta Haptic Feedback Glove that can be controlled and interact with my 
Apple branded headset over the NUOMIP — the New Millenntum Open Metaverse Interop 
Protocol, which also allows the system to access several “viewpoint orbs” which are 
secondary lidar cams mounted around my office to ensure that the space can be accurately 


modeled and rendered and persisted if my headset is offline. 


Multiple Cursors, Audio Privacy & Thwarting Lip Reading Attacks 
The seating chart now comes to life along with a virtual tool pallet that projects out at a 45 
degree angle from the top edge of the chart providing a class MacDraw inspired array of 
editing tools. A solid black cursor appears on the chart when I virtually tap it with the pen tip. 
At this point Mark’s copy displays a purple “Guest Cursor” depicting the position of 
my working cursor on his copy and as soon as he rezzes a virtual pen of his own and taps it 
on his copy of a chart, his “Guest Cursor” appears on my copy. Now we can take turns 
repositioning tables and chairs on the charts, we can also converse normally as anything we 
say is audible within the room, unless one of us turns to reply to someone outside the sim 
who is speaking to us in real life. The system filters out any such “Noises Off’ and 


immobilizes one’s avatar’s lips to prevent other sim visitors from reading them. 


Datum Extraction and Unfolding Private Contextual Spaces 

One of the names on the chart’s legend is unfamiliar to me, so I let go of the simulated chart 
and my simulated pen, which both hang in mid air as if not subject to earthly gravity. I then 
employ a “Pluck and Plop” gesture with my pinkie finger extended to draw a copy of the 
name out of the chart and deposit it at the top left corner of a 2-D Contextual Space that is 
initially invisible to Mark unless I say “Display This” or employ a “Wiggling the Fingers of 
Both Hands” gesture to put the virtual display in phase with my colleague. 


Commands spoken while facing such a display are treated as “Noises Off” interactions 
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with respect to anyone else in phase with one unless one has made the display visible to the 


room. 


Interrogating a Social Graph with Speech and Gesture 


I say, “I need a Social Graph cross referenced with the guest list extended once highlighting 
conflicts”. This produces a series of tag cloud like name clusters with headings like 
“Friends”, “Colleagues”, “Known Romantic Partners”, “Academic Rivals”, and several other 
custom headers reflecting the results of queries I had previously defined. Two names jump 
out at me in bold text with an edge connecting them revealing to me that one of the attendees, 
while not an “Academic Rival” of anyone on the guest list is the “Romantic Partner” of a 


non-attendee “Academic Rival” of another guest. 


I stab at these names with my index finger causing them to be highlighted on our 
seating charts where we see both guests tentatively seated at the same table. Mark is way 
ahead of me at this point and has already reconfigured two other tables so we can separate the 


potentially antagonistic guests to head off the risk of awkward moments. 


I dismiss the Social Graph with a “Flick of the Wrist” gesture and close my copy of the 


seating chart with a “Collapsing Clap” smooshing it into nothingness. 


Guide Mode & Next Gen Immersion 


I turn to Mark and say, “Let me show you the latest code I’m working on.” I put my hand on 
his shoulder which is an “Initialize Guide Mode” gesture (subject to override with an audio 
“No”’) and command “Goto to my NeoTech Codebase”’. 

I also invite Fabien to join us. He is testing the latest experimental full immersion rig 
that employs a series of robotic arms that extend and latch onto to his haptic feedback 
jumpsuit at key support points making it possible for him to lean agains or sit on virtual 
surfaces as well as to walk, run, jump, climb, kick, or move in virtually any direction. To an 
observer it would almost looks like he was being suspended by the tentacles of some Eldritch 


Horror, but from his perspective, everything in the virtual world would have full tangibility. 


The more conventional treadmill rigs Mark and IJ are using only support a casual 
walking pace, but are more than capable of letting us independently meander in any direction 
and reach out to interact with things. 


135 


Literate Programming in VR 

In any case, with a loud “Ping” Frode’s office cross-fades into an abstract data-space that 
looks like a modern dark theme museum gallery with a textured grey carpet floor and a 
number of smoked glass panels bearing luminous off-white text in my favorite coding font. 
The text itself is meticulously typeset for optimal line breaks employing micro typographic 
control. 

The main panel bears fully justified text at a relatively large optical size describing the 
plan of the program which has chapters (represented as adjacent panels mixing expository 
text with actual source code) devoted to topics like “Reusable Utility Functions”, “Error 
Handling”, “Analytics”, “Spatial Visualizations”, “Interactivity Affordances”, a “Command 
Language Grammar”, and finally an “Embedded Dataset”. The Chapter Panels are similarly 
decomposed into Sections and Sub Sections and arranged like a series of exhibits in a 
conventional Art Gallery with fragments of actual code set off in a Neon Green with Amber 


colored interlinear glosses providing data types and usage statistics. 


Quasi Natural Language Source Code 


The actual source code proper looks far more like text than the old notations popular at the 
turn of the century, although it uses terms-of-art and circumlocutions to represent computing 
concepts by name. While this is somewhat more verbose than using a traditional terse 
notation, it vastly enhances clarity by avoiding overloading common symbolic notations with 
conflicting semantics and encouraging a declarative style by supporting code like this 


function declaration: 


This is a function that takes 
the name of a file 
and a list of functions 
that take 
the name of a file 
and return 
a text 
or nothing 
and a list of words 


and returns 
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a list of 
lists of 
words and word counts 
or nothing 


or nothing 


Alternate Viewspecs & Hyper-Reality Seating 

Fabien rezzes in and says “Show me the Code”. This signifies to the system that he isn’t 
interested in the Literate Program view that serves as the space’s default; this causes 
everything in his view of the room to fade away to be replaced with a gently arced panoramic 
“Wall of Texts” depicting the linearized source code spread over seven columns and sized to 
fit all of the code in, with optimal line widths on each column. It also rezzes a control chair 
that Fabien sits down in (actually being supported by the tentacles of his VR Rig and the 
stiffening of the smart fabric forming his haptic suit — a stunning implementation of Hyper- 
Reality (Hickman, 2023)). 

His avatar’s opacity is reduced by 30% from our perspective whereas our avatars’ 
opacities are similarly dialed back to signify that he is partially out of phase with us. 
However, any selections made any of our views will be kept in syne with each other. So if 
Mark selects some code it will be highlighted in his spotlight color (for small numbers of 
participants the system simulates a different primary color spotlight shining on them in the 
shared space and uses that color to color code his cursors and selections) in Fabien’s code 


view. 


Once seated a control console rezzes in front of him with virtual buttons corresponding to 


filtering tags and custom views. 


Unfolding and Populating a Benediktine Cyberspace 
I say “Join Fabien” and crossfade into his Sub-Phase and view of the codebase. 

I then say to him, “Lets see the code fragments arranged by cpu-usage, length rounded 
to dozens of lines, and a standard deviation clustered revision count.” He opens a new 
visualizing by forming both hands into L-Shapes at 90 degrees to each other as if holding a 
box with his left thumb on the bottom and his left fingers on its left side with his right thumb 


on its front and his right fingers on its right side drawing it open to create a three faced open 
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cube. Then he plucks the labels for the three dimension I asked about and throws them onto 
the walls of the space where they hit with a “foop” sound and populate each axis with 
appropriate labeling. Seconds later a cloud of dots appears in the space forming several 


natural clusters. 


Rezzing a Vector Cursor 

I then say “Luminosity by instance count” causing several points to glow more brightly than 
others. Rather than flying through the data cloud, which might give me vertigo, I hold up my 
right hand palm out and say “Vector Cursor’. A disembodied virtual hand appears and I push 
my hand forward causing its virtual twin to move outward and into the visualization getting 
smaller to match its local perspective, I shift my real hand around causing the cursor to track 
the direction of my hand movement with its speed mapped to how far I moved my hand from 
its initial position, allowing me to naturally speed it up and slow it down until it reaches the 
region of interest where I point at and tap on a particularly bright dot representing 
infrequently edited code fragments between 12 and 24 lines in length with abnormally high 
CPU usage. 


Natural Language Codebase Interrogation 
I then verbally query “Do these fragments share any library calls?” A results panel rises up 
from the floor to my left with of 27 library candidate functions. “Exclude calls also made in 
fast code.” The system knows what I mean by fast in this context and filters out all but 5 


library calls all of which reference the same API. 


Fabien jumps in with, “Show Us the API Source” and our view crossfades to a montage of 5 
function definitions which are automatically color coded by function category with all five 


showing a bright red network data request to the same IP address. 


Domain Specific AI Agents 
“Ping that Server’, I say. And 4 seconds later Clara, the AI we use for code optimization 
replies “The server is showing a 2.9 to 3.8 second response latency. Shall I look for an 
alternative implementation of this API?” 

“Yes”, we both chime out in unison. 

A few minutes later Clara replies “Jake’s Faster Transformer 3.0” provides a compatible 
API after a unit scaling conversion. Shall I write a wrapper function to invoke it and splice in 


the new code?” 
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“Yes, Clara, then rerun the test suit and give us the benchmarked improvement data.” I 
say. 

The revised code appears alongside the original function calls with the changed line 
glowing brightly. 

Clara soon informs us that the new code will reduce execution time by about 38 


percent. 


Fabien and I bring ourselves back into full phase with Mark and agree to meet up again ina 
few days to write up a paper describing how working in XR has enabled us to find new ways 


of getting things done that wouldn’t have been possible in physical space. 
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An Early Manifesto for Spatial Computing 


Alan Laidlaw 


Spatial computing is a new material. Materials are more responsive to some design patterns 
than others. It makes no sense to say “Skyscrapers should be built out of cotton, if only cotton 
were strong enough.” It’s a better policy to figure out what design patterns play to the 
strengths of that material. 

Games will probably do most of the work of figuring out which design patterns fit best 


with spatial computing (and all its genres). 
Games are inherently more dimensional than textual interfaces. 


But text interfaces haven’t faired well in the last decade. One can’t help but feel 
overwhelmed by text, whatever its origins. Social threads, internal documentation, and 
countless tabs (of countless ad-riddled articles) are all varieties of textual interfaces. Synoptic 
comprehension and intuitive collaboration have suffered under the current hegemony of flat 


graphical text interfaces. 


Could text become less overwhelming if screens lost their dimensional constraints? Is 


XR the right material for word processing? 


I bet that conventions around text will have to change first. Multi-modal devices and AI 
augmentation blow apart current conventions about what writing is. Articles may not be 
written so much as composed into polymorphic shapes that complement whatever material 


they arrive at. 


We’re at a rare moment when several layers of abstraction are up for grabs. Just as 
we’ve been trained how to drive cars, we’ve been trained how to operate personal computers. 
The heuristics are, for the most part, locked in. Mobile phones are much younger than PCs, 
but it seems most of the heuristics are settled for phones as well. By comparison, XR is 
uncharted territory. 

Even though games, being multiplayer and high-dimensional, will probably define the 
heuristics of spatial computing, this is time for wild Hail Mary attempts to push the 
paradigms of personal reflection, public works of digital civil engineering, and possibly 


uncover latent talents in the material of the human body. 
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The Tapestry Project 


Bob Stein 


The reason we collect the past 
is to enable the present 


to prepare for the future 


Brewster Kahle, founder of the Internet Archive, posed a question: How could the largest 
digital library ever assembled make its vast collections—images, video, audio, texts, software 


and web pages—more useful? 


The Tapestry Project answers this question with a paradigm breaking new tool that 
anyone can use to construct a non-linear landscape of ideas and the connections between 
them — a new form of writing which allows readers to interact directly with the author’s 


ideas and the sources they are based on. 


Tapestries are NOT simply lists of links, requiring the user to click and go somewhere 
else. Rather they are an array of functional objects . . . each one visible and ready to be 


accessed and explored. Here’s an example. 


In 1968 James Baldwin and William Buckley engaged in an historic debate at the 
Cambridge Union. Based on the event, this tapestry consists of a dozen elements, each of 
which can be experienced directly and in its entirety. Readers can annotate on their own or in 


conversation with others. 

¢ The full 58-minute video divided into component parts 

¢ a2019 Atlantic article explaining the historic importance of the debate 

¢ a New Yorker piece about the March on Selma referenced by a student speaker 
¢ Wikipedia pages of both Baldwin and Buckley 

¢ The Fire Next Time, Baldwin’s most famous book at the time of the debate 


¢ Wikipedia entry for Buckley’s magazine, The National Review 
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Tse. “4 
debate . : az Proposition: 


7a BALDWIN ole “} 7 That the American Dream is at 
~ 7 ; = 


the expense of the American Negro 


Key Features of Tapestries 


1. Users create a Tapestry by dragging objects — books, articles, web pages, images, audio, 
video, software and entire web pages onto a blank infinite canvas. Size and placement is 
entirely up to the maker. 

2. Tapestries are hosted by the Internet Archive but displayable anywhere. Think of 
Tapestries as portable compound illustrations made up of fully functional digital objects. This 
is a screenshot of the Baldwin/Buckley Tapestry running in a WordPress blog. 

3. Tapestries’ design allows for a wide range of use. Naive users of any age will be able to 
make useful things with little instruction and easily share them with others. Experienced 
designers and coders will be able to create complex works, extending the open-source 
architecture as needed. 


4. Readers can comment on any part of a Tapestry or on the whole. Comments can be public, 


or posted to a private group. 


5. Assuming creators have given permission, readers can fork Tapestries and remake them as 


they see fit, by adding, subtracting or rearranging elements. 
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Example Tapestries 


Nine versions of a favorite Emily Dickinson poem, including one in Mad Magazine 


Best Movies of All-Time 
Once every decade since1952, The British Film Institute’s Sight and Sound magazine has 


published a list of what critics consider the best films of all time. The IA has a complete set 
of the magazines and it’s very interesting to trace the shift in the cultural zeitgeist over time. 
This tapestry includes the Best Ten list for each decade, the complete issue of the magazine it 


was published in, the accompanying story and the trailer for that decade’s #1. 


The Encyclopedia Project was a little known collaboration between Atari and Encyclopedia 
Britannica in the early 1980s. The tapestry consists of 500+ pages of internal documents, 
videos that inspired the project and a series of prescient drawings by Disney animator, Glen 


Keane imagining various ways an intelligent encyclopedia would be used. 


The Encyclopedia Project 
Britannica and Atari, 1980-1983 
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Unusual Cookbooks. A librarian creates a tour of her favorite cookbooks including videos 


she creates within the Tapestry Maker. 


Browsable Video. The Internet Archive’s unparalleled TV News collection is now browsable 
via visual scrolling. The seven panels in this tapestry represent the entire output of seven 
different Russian TV stations on June 3 2022. Each of the panels consist of hundreds of 
screenshots. Clicking on a screenshot opens the corresponding video which immediately 
starts playing. 


Learning BASIC. A high school teacher wants her students to learn about computer 
programming circa 1980. She drags an instance of BASIC running in an IA emulator and a 
kid’s programming manual into a tapestry which then becomes a workspace for students to 
complete the teacher’s assignment to make something. 


Synching text with Audio or Video. Sally Rooney’s article in the Paris Review about the 
importance of Ulysses was the inspiration for this tapestry which includes three different 
versions of the book, three Ulysses “guide” books, an audio version synced with the text and 


a charming video by an artist who drew a picture on every page of the book and then pasted 
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those pages on Broadway lamp posts 


Tapestry as Art Gallery This tapestry replicates the wall of a museum or art gallery. Each of 
the illustrations is a video of an electronic book by Alex Itin. 


Digital Book Shelf. The tapestry maker can import entire Internet Archive collections. This 
one, comprised of Atari Magazines suggests that Tapestries might be a key step in a long 
sought after goal — the browsable digital bookshelf which extends the serendipity of library 
stacks to the networked screen. 
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Coda 


Let’s call it. . . the print era is officially over. We now live in a networked digital world. 
As a result everything is breaking and everything is being born anew. 


We need all the understanding we can muster which is why the IA’s prodigious effort to 


collect as much of the analog and digital record as possible is so important. 


New forms of expression are emerging. We may not know the specifics, but two aspects 
are coming into focus — a) expository text is being dislodged as the principal medium of 
intellectual discourse and b) the creation and consumption of media are increasingly social 
activities. 

From this perspective, people need tools that foster collaboration — finally realizing 
the vision of the internet’s founders who understood that the problems facing humanity are 


too complex for individuals to solve on their own. 


Making new and useful knowledge is a collective task and we stand at a pivotal 
moment. Al-generated content will soon form a non-negligible percentage of the internet’s 
content, making it more important than ever for users to have ways of evaluating the source 
of everything they encounter. Tapestries present a model for moving through this dilemma: 
objects in a Tapestry are always connected to their sources, and are part of a network of other 
Tapestries and creators. Readers can use this network to make smarter judgements about 
what’s trustworthy. 


For more information on the Tapestry Project contact: 


Bob Stein Dan Visel 
futureofthebook@gmail.com dbvisel@gmail.com 
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Intermediatheca and intermetadata 


Antonio Sena 


‘Summary of the Catalog and the hypothetical Exhibition 
Historia de uma Intermediathece: 

Centro Comunitirio das Terras - umus 
Julhe/Agoste 2016 (original in Portuguese) 


© What is THIS? Explanation 


© Historia de uma Intermediatheca Catalog cover 


© Local 38°237916°N - 28°13'45.55°W - 2016 
Tetras, ha do Pico, Agores, Portugal, page 2 


@ Documentos do estado da ex-Escola Primaria das Terras 
‘em 2013 © mudangas efectuadas em 2016, page 3 


@ Imagens do interior da Exposicto, Estante @ Arquivo, page 4 


@ Projecto Centro Comunititio dedicado 4 Terra, 
420 Trabalho Doméstico @ a0 Trabalho Agricola 
interigados pela Informatica page S 


© [mitho x) e/ou (mitho * ») 
‘9 milhavel pelo milho do minade pelo mithar do mithador, page 6 


@ Ficha Herculane A ficha de qualquer coisa que tem, 
wmuitineamente, a parte © 0 todo, page 7 


@ Estante Topotogica Items colocados do local (b esquerda) 
para 0 global (& dieita), page & 


@ Estante Chronologica items colocados do mats antigo 
(b esquerda) para o mais recente (4 dweita), page 9 


©.0 que 6 ums Intermediathec? 
Metamorfoves informéticas, a caminho do ether, page 10 


© £0 que 6 1STO? Uma inica base de dados relacional 
‘para todas as coisas que cada um pensa que existe 
‘que no existe, page TT 


@ Exposicso # Catéloge Uma oxposicio @ catilogo 
[proporcionais ao local, 4 regilo, a0 pais, etc, Exempic: 
Uma amostra(zinha) de Miho, page 12 


© Cronologia das Tentativas para tomar publica 
‘a Intermediatheca entre 1988 © 2020, page 13, 


© Gibliografia 
conceito de Intermediatneca, page 14 


© Ficha Técnica ¢ Indice page 15. 


© Catalog back cover 


© toternet Archive Home page 
Public brary = archive.org 
The Internet Archive is a US-based digital 
brary that promotes “unwersal access to 
all knowledge" It provides public access to 
collections of digitized materials, ncuding 
websites, applications, games, music, movies, 
videos, sounds, photographs, as well as 
rmlions of books, journals and scientific 
iterature 
Established: May 10, 1996, San Francisco, CA 
Meadquarters: San Francisea, CA 
Founder: Brewster Kahle 
Prevdent Brewster Kahle 
Number of employees: 169 (2019) 
Source Wikipedia 


© Intermediatheca [betwoonsmodiatcase] 
Look at your smartphone, 
at your tablet, at your computer. 
‘They are “cases” with many media: 
words, sounds, pictures, videos, games, propaganda, things about many things But they are unorgansed, naccurate, even anonymous 
Now look around you - you are surrounded by many media about everything by mostly unknown authors 
Of course, they were not objectively chosen, selected and organized by you with specriic purposes 
‘The intermediatheca is 2 software that provides and promotes that organization and the opportunity to add and link everything you wish or need Just Uke the way you think 
Each person is responsible for his/her data, 


ODOUJOIPSWG4U] UO JO} BIOMJJZOS 


EPepeyouezUP) PLO OJOPOJeV\ 'OJOPOJOJ4 — eALYo oU@ZUY 


1202-8861 - S801NOS PUD sjVaUND0g ‘VoYO1SUOWIEG 
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Front and back cover of the Software for an Intermediatheca linked brochure, 2022 


https://archive.org/details/sena-antonio-software-for-an-intermediatheca-authors-edition- 


cancela-da-areia-2022/mode/2up 


This is the back of the bookshelf used at the Centro Comunitario das Terras - umus (CCTu) 


as an exhibition panel in 2016. 


The first item displayed on the exhibition panel is the 
éWhat is This? idea, 1988-2019 (in English/Portuguese): 


a single relational database for all things each one thinks exists - and the ones that doesn't. 


https://archive.org/details/sena-antonio-o-que-e-isto-what-is-this-ed.-self-published-edicao- 


de-autor-cancela-da-areia-2019 


The second item displayed on the exhibition panel is the 
Historia de uma Intermediatheca brochure, 2020 (in Portuguese). 


https://archive.org/details/sena-antonio-historia-de-uma-intermediatheca-ed.-de-autor- 


cancela-da-areia-2020 


The third item is where the first public prototype of the Intermediatheca took place: 
Centro Comunitario das Terras - umus - PROJECTO (CCTu) brochure, 2014 (in Portuguese). 


https://archive.org/details/sena-antonio-sena-ana-centro-comunitario-das-terras-umus- 


projecto-ed.-cctu-terras-2014 
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The remaining pages on display are an explanation of What is an Intermediatheca?: 


an interdisciplinary and multimedia archive of artistic, scientific and everyday knowledges, 
analog and/or digital, 

organised in a single database of any media and of any thing, 

fundamental and hyperlinked - the sources - for children to adult, 

made available (physically) by chronological order, 

of interest to the community, 

accessible with or without power, networked, offline and online, 


but using, whenever possible, renewable and sustainable energies and self sufficient 


practices, 


namely agricultural practices in an urban or remote context. 


Imagine a person, since childhood, with or without schooling, 
being curious and an autodidact, 


building their own career/knowledge whatever the subjects. 


Imagine a library (a multimedia library) that it'll be sufficient for a child to grow, 
studying with or without school, researching and divulging simultaneously, 


until adulthood and old age. 


At the bottom of the panel lies the catalog of the Intermediatheca's first and only public 
exhibition: 
Uma amostra(zinha) de Milho, 2016 (in Portuguese). 


https://archive.org/details/sena-antonio-cur.-aavv-uma-amostrazinha-de-milho-catalogo-ed.- 


cctu-terras-2016 


The last item in the exhibition is the Internet Archive Home page, 
https://archive.org 
through which it is possible to access almost all of the items I need for the Intermediatheca: 


https://archive.org/details/fav-luzetherpixsena 
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N.B:: 


1- Imagine that you can have access to the sources (no matter the media) of almost all 
knowledges, 


a minimum intellectual reserve of humankind, 
2- you can put them inside your tablet (if digitally transcribable), 
3- inside the same and unique relational database, 


4- and you can relate them, create hyperlinks and operate with them as you wish. 


Of course others will make different choices but with many similarities. 


You must be concern how much it occupies: the Essential Knowledge per MB (EKpMB). 


The first and second steps are already possible (I’ve done that: physical sources at home and 
their digital equivalents in the tablet). 


The third and fourth are not possible yet with the desirable fluidity (it needs more than one 
app and a lot work of mine). 


https://archive.org/details/sena-antonio-software-for-an-intermediatheca-internet-archive-file- 
picture-abstr_202311/mode/2up 
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‘Computer Narrative’. A view from the distant 


past... 


Rob Swigart 
NER/BLO (New England Review/Bread Loaf Quarterly) 


Tenth anniversary issue, autumn, 1987 


I believe this essay from 1987 still holds up. In 1986 Activision published Portal??, a project I 
insisted on calling an interactive novel. Around that time I was writing the stories that 
became a CDROM hypertext from Eastgate Systems called Down Time (2000). The 
interlinked stories bore titles drawn from a dictionary of computer terms: GIGO, For-Next 
Loop, The Glitch, Random Access, metaphors for the complexities of characters’ 
relationships in Silicon Valley. I called the short sections of text that interacted with one 
another NITS, for narrative units. Now they are commonly called lexia. One feature of Down 
Time was text on screen accompanied by narration. When I gave a demo of the project at a 
conference I sat in the audience. The interaction between visual and auditory reading amazed 
me: the eye would skip ahead, the sound lag behind, the ear would take in something the eye 
had already seen, then the eye would skip back. The two modalities reenforced each other in 


a way I hadn’t anticipated. 


Today we also have DRITS, Dramatic Units of video. The first of these I saw was from a now 
ancient interactive Apple Laserdisc project about the discovery of the DNA molecule, with 
navigation by way of chronological nodes on a double helix. Watson and Crick present the 
latest DNA model to their x-ray crystallographer, Rosalind Franklin. She's stunned. “It’s 
beautiful,” she says. No more than 30 seconds, but a very powerful harbinger of things to 
come. Many have come to pass, or are coming to pass. Text is visual, but represents meaning 


as well as sound. The visual encodes information. All modalities are in play. 
Note on Using the Computer to Write 


Two things interest me about writing on the computer. First, it allows text to go through a 
phase-shift from solid to fluid. Words and phrases, sentences and paragraphs flow and find 
their own level. Paper is solid, hard, intractable. Once finished, text may be frozen onto 
paper, but until then it is dynamic and alive on screen. Second, the computer as a new and 


very interactive medium has borrowed its language from other areas of human life. As 
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always, the process renews language, makes it metaphor again. And those metaphors reflect 
back on the people who use them in unexpected and revealing ways. This is the basis for the 


stories in "Down Time." 


Where They Come From 


Once upon a time computers were large, complex, and difficult. They were confined to 
airconditioned temples, and attended by a priestly elite, insiders in white coats. Computers 
could do one thing: they crunched numbers, ever more and more. 

Personal computers changed the world, though. They are small. They are cheap. They 


fit in the home. Such power can free the imagination. 


Computers still crunch numbers, of course. Folks in homes balance their checkbooks 
with them, or forecast their financial futures with spreadsheets. Computers can calculate the 
angles and vectors of spaceships on an arcade game screen as easily as the orbits of the space 
shuttle. 

Computers process words, too, of course: letters, memos, business plans, newsletters 


and Great American Novels proliferate across the land. 


And computers allow access to, and searching through, vast databases of knowledge, 
electronic storage bins of trivia, dumpsters for the world's cultural detritus. So far such 
databases are collections of specialized information, loosely organized, poorly distributed. 
But the process of searching and collecting grows daily easier and more powerful. 

Because they are computers, we know that what they tell us, the information they hand 
over and organize, is true. Computers, after all, are the oracles of twentieth century man. The 
knowledge they give us is real, true, and good. If they make mistakes, they are human errors. 
Someone entered incorrect data. There was a glitch. Human error. Computers, in and of 
themselves, are truthful. Their knowledge is authentic. 

But there is no particular reason why such knowledge must always be authentic. Such 
knowledge is text organized in sentences and paragraphs, after all. 


Just like novels. 
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The Medium 


A medium for narrative art has two requirements, without which it cannot exist: it must have 
a structured way of organizing the "information" it conveys, even if that information is 


fictional; and it must have an audience. 


Large mainframe computers of the 1960s had no appreciable audience. Those who 
would work with them were priests with a job to do. Their client base was small, wealthy, 
businesslike, and uninterested in art: moving the goods, the personnel, the payroll was 
enough. 

Personal computers are now widespread. They will become more so. They will appear 
in cars, appliances, stereos, telephones. Already millions have them. Most play games with 


them. They are the audience, waiting for art. 


Grunt and Gesture 


Art evolves from playing with technology: most technology communicates, either indirectly 
by shaping the environment through tools (chipped flint cutters, scrapers and hammers), thus 
determining the course of culture, and further the way the culture views the universe, or, 
more directly, by acquiring, manipulating, storing and retrieving information, as in language 
and writing. 

Communication takes many forms. It can be sound, or it can be sight. It can be grunt, or 
it can be gesture. All others evolve from these two, the grunt and the gesture. A caveman 


points at a heap of mammoth manure and grunts. He means, "Dinner went that way." 


Later he paints the mammoth on a cave wall. He is shaping the world. He is forcing the 
mammoth to cooperate. He is doing his magic on the mammoth, on his fellow hunters, on the 
wall of the cave. We think he was creating art. He was not, because he was not playing. He 
was acquiring, manipulating, storing and retrieving information. Important information, 
relevant to his survival. 

Later still he began to play, when the hunt was successful. He decorated things. He put 
his grunts together into pleasing patterns. He made song, and he made dance out of gesture. 
Art. Still, grunt and gesture. 

He began to organize his painting into more efficient, more complex forms. He made 
stereotyped images: pictographs, hieroglyphs, ideographs, small, graphic, visual bundles of 
idea or object, separate from the thing itself. Or his ox tilted and turned into an "A." The 
alphabet. 


When we play with alphabets, with written language, we create literature, increasingly 


complex arrangements of sounds, with rhythms, repeated music, content interacting with 


154 


connotation. 

When we play with visual, graphic information, we create painting, sculpture, 
photography, moving pictures, television: visual arts, gestures in the visible world, leaving 
behind a smear of meaning on our awareness. 

Grunt and gesture. A single grunt becomes two, four, eight, sixteen. On and on, 
increasingly complex, building words, sentences, novels, radio shows, rock-and-roll. A single 
gesture becomes Balinese dance, kabuki theater, the hula. Grunt and gesture together become 
opera, feature films, sitcoms, Shakespeare, Music Video. 

The computer combines text and graphics. We are learning skills at using the tools, the 
way sticks pounded on logs to send information over a distance became the blur of 
drumsticks in a punk band. The tool became a medium of artistic expression. 

With the skills we have in acquiring, storing, manipulating and retrieving information 


through the computer, we may begin to play. 
Games of course are play. Computers play games pretty well. 
Games are not art. 


Consider the nature of participation. We look for metaphors, analogies. What art forms 
have human beings developed in which two or more participate? Some kinds of 
improvisatory theater, perhaps. Dance, certainly. 

Could we consider a narrative art form that would be like dance? And how, then, can 


we create an environment in which a man or woman can dance with a machine? 


Narratives—drama, the novel, film, television, seize our awareness, pull us into other 
lives. They offer the closest we know of to true telepathy, to the experience of another as the 
center of the universe. They offer as well the experience of empathy, of feeling with those 
others, out there, the creations of another imagination, yet, while we experience the reading, 
the viewing, they are as real as we. More real. 

Reading is a collaboration, a dance, between the author and the reader. The novel forms 
a zone between the writer's imagination and the reader's involvement. The zone does not exist 
on the material plane, not the spiritual, but in the mystery we call mind. 

Computer artists, computer writers, must find that zone where telepathy may happen. 
Where a true dance can take place, where the player (user?) can act and fee! and see the 
results and consequences of that action. 

Compact Disk Read-Only Memory (CDROM, or CDI for interactive ROM) will offer 
writers and artists an incredibly powerful medium for these new kinds of drama to happen. 
We can only begin to imagine the impossible worlds that will suddenly appear through the 


window of the computer screen. 
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Imagine too the fine arts that may appear, combining textual poetry or story with visual 
and graphic movement, with music and sound, with interaction. We may begin with the old 
medium, music or the player's presence altering the colors and patterns on the screen, 
changing the sequences of words or stanza-equivalents, altering the emotional tone of the 


music, which changes the colors and shapes, the rhythms of motion, the qualities of the word. 


Computers will evolve an art form, an entertainment form. It may well be similar to the 
forms of computer work: databases, perhaps — fictional databases, databases that represent 
imagination, play. Databases that combine the grunt and the gesture, the graphic image with 
textual content. Databases with style, characterization, story, conflict. The flickering CRT 


screen may take us back to the cave, when we sat around the fire, telling stories. 


From Agriculture to Ikebana 


Narrative art is the closest thing we have to telepathy and empathy, to participating in the 
experienced life of another. The words, their arrangement and style, the way they organize 
images in our minds, the sequences of event and character shaping drama, give us leave to 


leave our bodies, to become someone else for a time. 


The medium must work for its art. Always we must ask ourselves questions when we 
try out a new tool. Do we listen? Is it dark when we do? Do we see? Does the light flicker 
and cast shadows? Do the words carry deep rhythms that enchant us? Do we participate in the 


art form? What is the nature of the participation? 

The computer was a tool. We used it to do work, to communicate wishes, goals, needs. 
It is a highly participatory medium, like speech. 

What other art forms might have such audience participation? Certainly written 
literature involves a collaboration between reader and author, but perhaps dance comes 
closer. 

We move from tool to game to art. This is the process, always. We can think about 
agriculture. 

Agriculture is a tool. Horticulture is a game, Ikebana is an art. The medium (growing 
things) informs the message (food, beauty, meaning.) 

The computer is a machine for processing textual and graphic information. It does word 
processing, financial forecasting, database storage. It is a tool. Databases store information in 
"fields." Spreadsheets put numbers and labels in "rows." These are the terms of agriculture. 
Old metaphors both inhibit and liberate us. The medium informs the message. 

What does this mean to narrative art, to storytelling? This century has discovered the 


nonlinear in life. Everything is relative. Heisenberg has given us Uncertainty. What you see 
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depends on where you stand. Quantum mechanics means the universe is at its very 
foundation binary, discrete, digital and discontinuous. There are no smooth, easy transitions, 
no causal connections. In this sense the computer itself is a metaphor. The best metaphors are 


media themselves. 


We till the computer fields (agriculture). We grow things for fun in those fields—Space 
Invaders, Zork (horticulture). This is play. Now let us make art. Ikebana: looking for the 
subtle, for rhythm and balance, for shade and color, for human interaction in a human world. 
Ikebana, Japanese flower arranging, uses the natural material of leaf and twig, blossom and 
ceramic within and against setting to make an aesthetic statement about symmetry and 
context. The statement is self-referential, pointing to the viewer's connection with and place 


in the universe at large. Ikebana is three dimensional haiku. 


We move through our own lives, our own cultures, and collide, swerve, think this, see 
that, feel something else, all at once. We rearrange constantly. We process information. We 


create ourselves. 


Computers sit now in networks, a democratic non-hierarchy. Nodes are distributed 
everywhere: we are all nodes. We are no more than five acquaintances away from anyone 


else in the world. Put 30 people in a room and at least two will have the same birthday. 


Narrative has moved from acoustic space (blind Homer at the fireside speaking into a 
three dimensional spherical space) to two dimensional linear space (typescript on the 
rectangular page, four walls of the monk's cell, silent reading as the eye moves from left 
to right, or from top to bottom, or from right to left, but always in lines, taking in the smaller 
lines of letters or ideographs) to multidimensional information space where hypertext can be 
layered, cross-referenced and glossed, graphics can expand and contract, flow and change at 
will. Constantly new constellations configure. We can point and move, command change as 
we wish. 

Information space is nonlinear, discrete, binary. Anything may connect to anything else, 
in any order, at any time. This could be chaos, of course, purely random mingling of data, or 
mangling of data. It could be meaningless. Or it could create new surprises, new insights, 
new meanings. It could tell us something of the nature of the universe, or at least what we 
now believe about the universe, which is in many ways the same thing. Our act of seeing the 


world creates its meaning. Now we have a new tool for seeing: information space. 


Information space informs the message. 
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Flowchart Narrative 


One possible basis for computer narrative structure is the flowchart. Designed for work 
(naval project tracking for construction of nuclear submarines, a complex and time-dependent 
project. Sound familiar? Like a novel?), this kind of flowchart allows branching, flowing 
away and back, network structuring with Time's Arrow always pushing ahead (we still 
experience in some form of linear time). A computer narrative allows for different kinds of 
experience for different (users). Everyone may access in a different sequence. We move 


through an information matrix as through a real three dimensional forest. 


Start can lead to A and/or C. C can lead to F and G, and so on. If Start is the beginning of the 
story, and A, B and C are consequences not necessarily bound into chronological order, or the 
cause and effect (characters go in different directions, for example), then the (user) may 
follow C, not B and thus go to F and G, and on to I. Each box contains a narrative unit, like a 
stanza in a poem or a section in a novel. 

Or some boxes may contain units of contextual information necessary to subsequent 
story developments. 

Traditional narrative contains many kinds of information. Character, context, history, 
emotional state, bits of local lore. The computer may treat these kinds of information as 
discrete "bits," semiindependent units. 

Or we can see opportunities for parallel development, for moving backward and 


forward through the story. 


A Byte of Information Space 


The universe remains, as always, fundamentally binary. Either we can know an electron (says 
Heisenberg) is there, or we can know it is moving at such and such a speed in such and such a 
direction. We cannot know both at once, though. A gate is either open or it is closed. We are 


either alive, or we are not. 
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In such a universe, though, there is always room for surprise: the electron has some 
small probability of being on the other side of the wall, an impossibility in a linear universe. 
In a Josephson junction (a neat trick of superconductors!) the electron will he somewhere it 


should not be. I can make a jump. In our computer novels, I can truly make a quantum leap! 


Possible also are neural networks, structures in which every narrative unit is connected 
to every other, weighed to determine how powerful any single transition might be, under 
flexible circumstances; the weights determine threshold levels for continuing. Neural 
networks are hierarchical, probabilistic, holographic. Narrative effects could become 
infinitely subtle. 

The smallest unit of information is the bit. On or off. Here or there. An electron either 
present or not. Bits arranged in different ways have different meanings. Eight bits (a byte) 
may be arranged in 256 different ways. For 32 bits (the fundamental number of the newest 
generation of machines) the number of possible combinations is very large indeed. We 
approach the complexity of life itself. 

(The brain has 10 to 20 billion neurons; each neuron has up to 100,000 connections 
with other neurons. This creates a very large network, with many possibilities for 
mathematics, poetry, art, aggression and defense, cuisine and catastrophe.) We may never 
reach such complexity (after all, then there would be no art, since there would be no editing, 
no limitation, and we would have life, not art at all.) 

But the computer uses text. Linear, you say, letters marching across the screen, line by 
line. Yes, our old metaphors still constrain us. We still use our eyes, though increasingly 
computers will let us use our ears as well, and our voices, our hands, even our good taste! 
Text builds mindpictures, dreamlife in our heads. Information processing liberates those 
mindpictures from the "real" world. Bits flow, jump, combine, separate, recombine, endlessly, 
making new texts. The mind imposes order, but the frontier has moved out again. 

The packets of information (bits, bytes, data, information) are discrete. They too flow, 
jump, combine, separate, recombine. This is narrative as flowchart, flowchart as narrative. 
And poetry. There are now visible spaces between the packages, the moments of silence 


between the meaning. Moments of silence which are the meaning. 
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A Bit of Reality 
Computer narrative is just beginning to discover its form. 

For the moment it may mean that the most appropriate narrator for a computer narrative 
is the computer itself, presenting the story in its "voice." This is selfreflexive: the computer 
tells the story about a computerized world through the medium of the computer. Here is no 
pretense, though, that the computer is something else: a movie, or television. And, in time, we 
will move away from this restriction. When computers are as much a part of our lives as 
television or the telephone. 

Much of a story's information may be graphic, animated, aural; the computer can 
handle it all, presented as pixels, as sounds. Charts, graphs, plots, color coded, quickly seized 
by the mind and mind's eye, all make up the new formal structures of a narrative. A 
character's intelligence, general health, emotional state, looks at age ten? Imagine a window 
that appears to display such information; immediately you, the audience (reader, [user], 
player), desire it. Expand on an unfamiliar term, a bit of landscape, an object: the computer 
can display an encyclopedic entry on a term, define a type of music, say, or give background 
on a piece; it can offer a guide book entry for a location, interesting local sights, good 
restaurants, items of local culture. 

Such things are possible now, determined only by the storyteller. Hypertext can do it; 
soon compact disk memories will offer greater detail, more extensive searching. The search 
will become part of ii( enjoyment. For now, we must search the near-term nature of the 


medium itself. This should keep us busy for a few more years. 
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Mintter, an open hypermedia system for the web 


Alexandr Burdiyan, Horacio Herrera, Julio Garcia, Eric Vicenti, Gabriel Herrero-Beaumont 


Hypertext serves as our primary medium for thought and communication. With the advent of 
the web, it encapsulated the capabilities for these functions, simplifying the complex 
hypermedia systems developed from the '60s to the '80s. The web emphasized networking 
and decentralization, democratizing information access and winning the hypertext race. 

However, in its current state, the web falls short of realizing its potential as the ultimate 
medium for thinking and communication. It has led us to rely on centralized platforms like 
Twitter and Instagram, driven by central planning. Unfortunately, these centralized structures 
struggle to contend with the intricate dynamics of our digital societies, stifling collaboration 
and innovation. 

Our concerns extend beyond technology; we are deeply troubled by the erosion of 
community collaboration, the constitution of knowledge, and the hijacking of the editorial 
process in journalism and academia by colossal monopolies. These issues strike at the core of 
our democracies and the scientific process, prompting us to embark on a mission to reshape 


the digital landscape for the better. 


In response, Mintter Hypermedia emerges as a solution, aiming to complement the web 
by enabling collaboration within the framework of decentralization and networking. We draw 
inspiration from legacy hypermedia systems, particularly those designed with 
decentralization and networking in mind, such as Augment and Xanadu. Additionally, 
projects like the Open Hypermedia System from the University of Southampton, conceived in 
the era of the web, have been instrumental in bridging the gap between traditional 
hypermedia systems and the modern web. Notably, insights from hypertext experts like Jakob 
Nielsen and Mark Bernstein have enriched our perspective. 

Mintter Hypermedia is an open publishing system that enables communities to 
collaborate on structured and deeply linked content. Within the Mintter ecosystem, all content 
is cryptographically signed, versioned, and made permanent with IPFS. Key features include 
signed authorship, verifiable version history, an open editing philosophy, and direct peer 
connections. 

Authentication in Mintter is designed to facilitate multi-device usage seamlessly. Users 
are given a secret key, a 12-word phrase from which the Mintter account key is derived, and 
will serve as their unique identifier and ensure a consistent link between their identity and the 
content they create. Furthermore, Mintter employs a device-specific key to enhance security 


and enable multi-device access. All user content is signed with these keys, enabling third 
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parties to verify authorship, even in a multi-device environment reliably. 


Direct connections between users' devices over the internet eliminate the need for social 
media gatekeepers, offering users greater control over their content. This allows users to 
connect to their peers by simply sharing connection URLs. The underlying Libp2p 
technology enhances the efficiency of these direct connections and ensures that users have 
the autonomy and flexibility to interact and collaborate directly, bypassing centralized 


intermediaries. 


The Mintter Hypermedia System fully embraces content addressability by leveraging 
IPFS, ensuring data integrity and permanence. This p2p hypermedia storage base is crucial to 
fulfilling the vision of completing the web as a genuinely open hypertext system. 

In Mintter, our Entities and Changes system allows for the evolution of content over 
time. Our current Entities are Account Profiles, Documents, and Groups. Changes are 
immutable signed data. We employ Conflict-Free Replicated Data Types (CRDTs) to interpret 
these changes in the current Entity State, ensuring consistency in a distributed network of 
computers. Plus, we maintain a version history for all Entities, enabling users to track and 


revert to previous states, facilitating collaboration and knowledge preservation. 


The Document Entity in Mintter is designed to support the arrangement of blocks in a 
hierarchy, with many block types such as Paragraphs, Headings, Images, Videos, or Files. 
The system also enables the embedding and linking content through references to entities and 
change IDs, providing verifiability and independence in content referencing. 

The Mintter system fosters robust collaboration through rich and deeply linked content. 
Users can seamlessly embed or link additional content into their documents by referencing 
Entities and Change IDs. Content addressability assures that users can verify the exact 
version of the embedded content they wish to reference. Furthermore, users can archive the 
content they embed or reference. In addition, Mintter's application capabilities include 
tracking citations and back-links associated with any document, facilitating effective content 


management and navigation. 


Groups in Mintter are collections of documents, each with designated owners and 
editors. These groups can be cloned, facilitating changes in ownership. The Mintter 
ecosystem extends beyond individual collaboration, as groups can be published on websites, 
requiring a server and domain name. This approach ensures the high availability of important 
content even when peers are offline. 

In conclusion, Mintter Hypermedia represents a groundbreaking initiative to bring an 
Open Hypermedia System to the Web, underpinned by open standards. Our vision is a web 
governed by decentralized online communities rather than centralized systems controlled by a 


few administrators. With the technology we have at our disposal, we are poised to create the 
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literacy machines necessary for a new digital age. 


mintter.com hyper.media 
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Narratives of Collapse: Al, Climate, and the Future 
of Text 


Mez Breeze 


Introduction 


In the grand theatre of technological advancement and environmental change, two agents are 
currently playing out dramatic roles: the first being Artificial Intelligence (AI),with the 
second being our planet's climate. Both are complex, dynamic systems facing a form of 
collapse that will have profound implications for our future. 

AlI-the prodigy of our current digital age—is at risk of experiencing a phenomenon known as 
model collapse. This term (while perhaps unfamiliar to some) sends a shiver down the spine 
of AI researchers and practitioners: it refers to a situation where after an AI Model is trained 
on a vast amount of data, it starts to spit out a limited set of outputs and loses its ability to 
generate a diverse range of responses. This is akin to a once eloquent speaker suddenly 
becoming monosyllabic. 

On the other hand, our planet's climate (the life-support system for every organism on Earth) 
is undergoing what scientists refer to as climate collapse. This is characterized by a rapid and 
potentially irreversible change in global climate patterns primarily driven by human activity. 
The effects of climate collapse are already evident in the form of more frequent and severe 
weather events, rising sea levels, and disruptions to manifold ecosystems. 

Knee-deep in the Anthropocene, the parallel crises of climate change and model 
collapse in AI are deeply entwined with the future of text and discourse. As the fabric of our 
digital and ecological environments fray, our language adapts, generating new terms and 
frameworks for understanding and discussing such epochal shifts. 

In the environmental sphere, the term global boiling reflects the escalating severity of 
climate change. This lexicon (along with terms such as climate crisis, climate justice, and 
climate refugees) is still far from becoming commonplace: though rest assured it will soon 
shape the narrative of our times. These phrases capture the urgency of the moment and the 
need for a substantial and immediate paradigm shift—both in thought and action. 
Concurrently, AI's potential model collapse presents a type of parallel disruption along with 
AI model’s ability to hallucinate (presenting fanciful information as fact). Due to the 
phenomenon of model collapse, AI systems are now at risk of becoming increasingly 
unpredictable. This uncertainty threatens the integrity of Al-generated content which plays an 


increasingly significant role in text production. Such disruptions are exemplified by the 2023 
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SAG Actors and Writers' strikes. These strikes underscore a broader societal debate over the 
role of AI in creative fields, as well as highlighting the struggle to reconcile technological 


progression with artistic integrity and human value. 


The Enigma of Model Collapse 


Once the stuff of science fiction, AI is now an integral part of our daily lives. From providing 
extensive services to producing and editing text and code (think: Google’s Bard and 
OpenAI’s GPT4) and generating imagery and videos, AI has woven itself into the fabric of 
our digital existence. At the heart of these AI systems are models: complex mathematical 
constructs that learn patterns from data and use these patterns to make predictions or 
decisions. 

As AI models become increasingly complex and integrated into our daily lives, the risk 
of a catastrophic failure increases. This risk is exacerbated by the fact that these models are 
often black boxes with internal workings that are difficult to understand and/or predict. 
Coupled with the recursive nature of many AI training processes, such opacity can lead to 
model collapse. 

Model collapse occurs when a machine learning model begins to output a restricted 
variety of responses which diminishes its capacity to produce a broad spectrum of diverse 
results. This phenomenon is particularly prevalent in models trained on their own generated 
data, leading to a recursive feedback loop that exacerbates the collapse. A paper titled "The 
Curse of Recursion: Training on Generated Data Makes Models Forget" by Shumailov et al. 
posits that the use of model-generated content in training causes irreversible defects in the 
resulting models, resulting in the loss of the original content distribution's less-common 
outcomes. 

The implications of model collapse are far-reaching, particularly as AI begins to embody 
aspects that will shape the future of language and discourse, with AI models increasingly 
being used to generate human-like text. Such text can power everything from chatbots to 
automated news articles. If these models were to collapse, the diversity and richness of this 
generated text would be severely compromised, leading to a homogenization of digital 


discourse with far-reaching implications for our information ecosystems (and beyond). 


The Looming Threat of Climate Collapse 


As we grapple with the enigma of model collapse in AI, another form of collapse is unfolding 
in the physical world. Our planet's climate— a complex system that has supported life for 


millions of years— is undergoing a rapid and potentially irreversible change. Known as 
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climate collapse, this phenomenon threatens the very fabric of life on Earth. 


Climate collapse is characterised by a rapid shift in global climate patterns which lead 
to more frequent and severe weather events, rising sea levels, and disruptions to ecosystems. 
The Intergovernmental Panel on Climate Change has warned of catastrophic consequences if 
we fail to limit global warming to 1.5 degrees Celsius above pre-industrial levels. The 
potential for societal collapse due to climate disruption is a topic that has been explored in 


depth such as in Professor Jem Bendell's influential paper on Deep Adaptation. 


The parallels between model collapse and climate collapse are striking. Both are 
systemic issues that emerge from feedback loops — in the case of model collapse, the 
feedback loop between training and generation in AI models, and in the case of climate 
collapse the feedback loops between human activities, greenhouse gas emissions, and global 
warming. Both forms of collapse also lead to a loss of diversity: in AI models, a loss of 
diversity in outputs, and in relation to the climate, a loss of biodiversity including the risk of 
mass extinction(s). Both model collapse and climate collapse represent existential threats to 
their respective systems. In the case of AI, model collapse threatens the utility and reliability 
of AI systems, potentially rendering them useless for many applications. In the case of 


climate, climate collapse threatens the very habitability of our planet. 


Parallels and Lessons 


As we delve deeper into the phenomena of model collapse and climate collapse, the parallels 
between these two crises become increasingly apparent. As stated, both are systemic issues, 
emerging from feedback loops and leading to a loss of diversity. Both represent existential 
threats to their respective systems, and both are human-made crises, born out of a pursuit of 
progress and efficiency. 

In the realm of AI, our desire for more powerful and efficient algorithms has led to the 
creation of models that are beyond our ability to fully understand or control, with a tendency 
for prioritised performance and scalability over transparency and interpretability. Similarly, in 
the case of climate, our relentless pursuit of economic growth and development has led to the 
overexploitation of natural resources and the emission of greenhouse gases. We have 
continually prioritized short-term economic and humancentric gains over long-term 
sustainability, leading to catastrophic climate destabilisation. 

These parallels offer valuable lessons for how we approach progress and development. 
They highlight the need for a more balanced and sustainable approach, one that prioritises 
enduring stability over transitory gains and valuing diversity and resilience over efficiency 


and scale. 
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In the realm of AI, this could involve developing new training methods that avoid the 
use of model-generated data, as well as the implementation of mechanisms to ensure 
diversity in model outputs. It could also involve prioritising clarity and comprehension in 


model design to ensure that we can understand and control the systems as they are created. 


In the case of climate, this will require a radical reduction in greenhouse gas emissions, 
along with efforts to adapt to the changes that are already occurring. It will also require a shift 
in our economic systems and humancentric values, including pivoting away from the 


relentless pursuit of growth towards a more balanced and sustainable development model. 


The Ramitications 


The advent of large language models (LLMs) has revolutionized the field of natural language 
processing (NLP). These models, trained on vast amounts of text, can mimic human- 
generated writing and respond to user input, enabling conversations and customized output. 
But as we delve deeper into the implications of model collapse in AI systems, it becomes 


clear that this technology's future is not without its challenges. 


One of the most significant concerns is the potential for AI models to generate false or biased 
output. At present, LLMs still lack the ability to judge the reliability and relevance of 
information. This limitation can lead to the propagation of misinformation and perpetuation 
of existing biases, especially when dealing with complex topics. The energy consumption 
associated with training and running these computational models has become a serious issue. 
The climate impact of AI and NLP research is increasingly under scrutiny, with a growing 
focus on efficient NLP. The energy demands of these models (both in training and inference) 
contribute to carbon dioxide emissions and other environmental impacts such as water use 
and soil pollution. 

In the realm of text production, these challenges could manifest in several ways. For 
instance, the potential for model collapse would lead to a decrease in the quality and 
reliability of Al-generated text. This would have implications for various sectors, from 
journalism and academia to entertainment and social media, where Al-generated text is 
increasingly being deployed. 

The environmental impact of AI models could lead to changes in the way we approach 
text production. As we grapple with the reality of climate change, there should be a shift 
towards more sustainable practices in AI and NLP research. This could involve the 
development of more energy efficient models, changes in the way models are trained, or even 


a reduction in the reliance on AI for discourse. 


In the context of XR (Extended Reality including Virtual and Augmented Reality) these 
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challenges could have even more profound implications. A collapse in AI models could limit 
the potential of XR, affecting everything from virtual meetings and online learning to gaming 
and virtual tourism. On the other hand, the environmental impact of AI could lead to a re- 


evaluation of the sustainability of XR technologies. 


Moving forward, the language we use to discuss climate change must reflect the severity of 
our situation. Terms like climate emergency (as opposed to the much more benign climate 
change) and global boiling (not the more neutral-sounding global warming) more accurately 
describe the dire state of our planet. This shift in terminology is not just about semantics; it's 


about awakening a sense of urgency and a call to action. 


Innovation in sustainable practices and investment in green technologies are critical in 
reducing our carbon footprint and transitioning to a sustainable future. Economic policies 
must support this transition, with investments that prioritize long-term ecological health.The 
challenge of limiting global warming to below 1.5°C requires a radical overhaul of societal 
behaviours and international policies. Exploring various economic pathways and their 
environmental impacts is essential to understanding how to meet our climate objectives. This 
includes examining the potential of both reduced growth and degrowth strategies and their 
roles in a sustainable future, as well as embracing language that conveys the true scale of the 


climate collapse we face. 


Actions and Adaptations 


The parallels between model collapse and climate collapse are not just academic curiosities. 
They offer us a map for action: a guide to how we can prevent or mitigate these impending 
crises. The lessons we learn from such parallels can help us navigate our uncertain future. In 
the realm of AI, preventing model collapse will require a fundamental shift in how we design 
and train our models. We need to move away from the use of model-generated data in 
training and focus on developing training methods that ensure diversity in model outputs, 
centring on lucidity and comprehensibility. 

One promising approach is the use of 'adversarial' training methods where models are 
trained to generate outputs that are indistinguishable from real data. This can help ensure 
diversity in model outputs and prevent the feedback loop that leads to model collapse. 
Another approach is the use of interpretability techniques which aim to make the internal 


workings of AI models more understandable to humans. 


In the case of the climate, preventing collapse will require a massive shift in our energy 
systems: away from fossil fuels and towards renewable sources like wind and solar. It will 


also require changes in our agricultural practices (think: away from monocultures and 
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towards permaculture-based systems), our transportation systems, and our consumption 
habits. But reducing emissions is only part of the solution. We also need to adapt to the 
changes that are already underway by building resilience into our infrastructure, changing 
damaging and unsustainable agricultural practices, (re)learning (and emphasising through 
policy adoption and implementation) indigenous climate knowledge—all while developing 
new adaptive and sustainable technologies. 

The parallels between model collapse in AI systems and climate change extend beyond 
their shared characteristics of being systemic issues. They also have potential implications for 
the future of text, language, and discourse. As we navigate these challenges, it is crucial to 
consider not only the technological aspects but also the environmental and societal 
implications. The Future of Text Symposium emerges as a pivotal arena for addressing such 
challenges. By catalysing dialogue around the confluence of our climate crisis, AI 
development, and text production, the Symposium can foster a vision of a harmonious 
coexistence between technology and language—a future where innovation is not at the 


expense of our planetary or digital ecosystems but that instead enriches both. 
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A short note on writing letters : technologies and 


practices from Sumeria to sometime soon 


Livia Polanyi 
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Before turning our attention to how text might be composed in Extended Reality, let’s begin 
by taking a look back at some of technologies that preceded this latest way to record 
language in written form beginning with the initial development of written language in 
Mesopotamia where cuneiform syllabic forms were impressed into clay with a stylus. 
Scribes, the original literate knowledge workers who were trained to encode and decode text 
were also trained in a number of other technologies to be able to practice their skills — 


particularly the production of the clay tablets they inscribed their pictographic and syllabic 
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symbols onto as well as the fashioning, selling and repairing the styluses used for inscription. 
None of these technologies, except for the instruction into the mysteries of writing language 
were relevant to other ancient writing practices whether by the Ancient Chinese who began 
by inscribing pictographs onto bone but later progressed to using brushes and ink to record 
characters on silk and paper or the Egyptians who used ink on papyrus scrolls. The later 
Greeks and then Romans used similar technology until the introduction of the use of vellum 
or parchment and the codex, essentially sheets bound on one side like modern books. These 


remained standard until the 15th Century when the use of paper became widespread. 


Although literacy was relatively widespread among the upper classes in Ancient Greece 
and Rome and lower class Roman men were able to read and inscribe the graffiti that covered 
the walls of buildings in the capital, the key technology in use throughout the Ancient World 
and extending into the Middle Ages and beyond was still the scribe: a literate human who 
was employed to write and to read. Scribes knew how a document should be composed to be 


appropriate to the nature of the communication involved: to whom, from whom, about what. 


In early 20th Century America, for example, scribes were still very much in use by 
immigrant and homebound analphabeticswho needed to communicate with loved ones.And it 
is still the case that most of us rely upon knowledgeable amanuenses to prepare legal and 
financial documents including wills, property transfers, leases and so on. And this is because 
a key technology possessed by some and not by everyone is literacy, the ability to read and 
write. 

Today, while you, the reader of this text, as well as I, the author, would describe ourselves as 
“completely literate” and might bristle at the slightest innuendo that that is not the case, 
almost all of us are, in fact, woefully incapable of writing even a simple legal contract or 
even reading with full understanding a complex one. So, as we consider the history of the 
technologies of writing, we must consider as well the spread of literacy. Who could read and 
write a letter or simple text? Who could read or write a complex text such as a novel or legal 
contract or even a convincing apology or love letter? 

For much of antiquity, literacy was restricted to scribes and male members of some 
social elites including those entrusted to conduct and explain religious rites and very wealthy 
members of socially prominent families. Whilewe know from the pervasiveness of graffiti in 
Rome that Roman citizens often were able to process very short political or pornographic 
writings, in Europe, the ability to read and compose more extended, complex texts was not 
widespread even among members of the gentry until the introduction of the printing press at 
the end of the 15th Century and the resulting availability of books among other types of 


printed materials. 


At that time, education was expanding as well with rates of literacy among men rising 
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from 10% in England in the 1400s to 20% in the 1500s and 50% by 1750 when the 
percentage of literate women reached 40%. By the middle of the 17th Century, postal systems 
were in place both in Britain and Europe for both public and private correspondence. By this 
point, the Golden Age of Letter Writing by educated members of the middle and upper 
classes was well established and ladies of the leisure classes especially devoted time each day 
to their post, keeping friends and family up to date with details about their households and 
their own thoughts and feelings. The writing instrument during this period was the quill pen 
which underwent a number of design improvements until it was replaced by the metal pen in 
the early 19th Century. By the 1880s, the introduction of the fountain pen greatly reduced the 
tedious process of continually refilling the ink reservoir in the reeds of earlier pens whether 
feathered or metal. The repertoire of pen types continued to expand with the introduction of 
the “biro”, the first practical ballpoint pen in the late 1930s and then with the felt tips and 


other markers in widespread use today. 


But even before the time when the technology for writing by hand underwent a 
revolution with the introduction of the fountain pen, a far more radical era was beginning: the 
development of machines other than printing presses to facilitate the production of written 
documents. The first practical typewriter, invented and patented by the American Christopher 
Latham Scholes in 1868 was available for sale in 1874 while the first machine to record voice 
for later transcription, Alexander Graham Bell’s Dictaphone was invented in the late 1890s. 
Together, these two innovative technologies would transform American business document 
preparation when employed along with the ancient human scribal technology — in this case 
almost always a woman -- who would use a special speedwriting code to transcribe the text 
dictated into the dictaphone by her (male) employer before translating her notations into 
standard written English by pounding on the keyboard of the office typewriter. That 
keyboard, the QWERTY keyboard patented by Scholes in 1878 and still in use on modern 
computer and mobile phone devices today is familiar to all of us: the three rows of numbers 
with numbers on a top row above them. The letters on the top row spelling out QWERTY 
with the Q far removed from the U which necessarily follows it in English words. This design 
improved upon an 1857 design by Samuel Ward Francis and used in Scholes’s earlier 
typewriterin which a linear arrangement of keys each connected to a mechanism was used to 
print specific letters was used for text entry.The affordance of the linear arrangement for 
stringed percussive instruments which allowed for multiple keys to be pressed 
simultaneously was of no use in text entry for alphabetic languages — especially those like 
English which do not use diacritics --where keys inscribing multiple symbols at the same 
time are never needed for normal text entry (though they have proved useful in certain 


limited cases such as court stenography and braille entry that require specialized training in 
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linguistic transcription systems devised for those applications). 

But as familiar as we are with keyboards in which we use our fingers to pick out 
specific letters used to assemble words onto paper or digital documents, it is important to ask 
if the affordances of this 19th Century technology should determine the methods of rendering 
authors’ words into text in Extended Reality which will soon replace the physical 
environments in which we now work and communicate with others at a distance. Oddly, 
enough, it seems that VR designers thinking about text creation in ER do not seem to be 
asking this question. Currently, there are a myriad of clumsy proposals for having users type 
on imaginary keyboards in the air or on a physical surface in front of them while, in the ER 
world in which they have projected themselves, images of hands moving around images of 
keyboards pick out letters. They are missing the point of what is involved in “writing”; it is 
not the printing of individual letters or syllables but transformation of the words of a 


language into a written form. 


Although throughout the millennia alphabetic and syllabic symbols have been individually 
inscribed on some real or virtual surface where they are assembled into words, producing text 
involves inscribing an author’s words, not individual symbols which must be assembled to 


compose written words. What is needed therefore are technologies for producing written 


ys 


words, not individual written letters or syllables. 


Rather than developing technology to support weird hand choreography to pick out 
individual letters, ER technologies that can input words are needed. And we do have this type 
of technology.Speech understanding has been under development for decades and, while far 
from perfect now, we can expect great improvement in the relatively near future. When 
combined with AI Editorial Conversational Assistants, editing documents using speech could 
easily be done. Privacy concerns that might inhibit users from speaking aloud can be 
addressed by adapting devices from today’s disabled interfaces that transmit subvocalized 
speech for use in transcribing spoken words no one can overhear. The objection that users 
won’t know how to use these newer methods of input can be addressed through training and 
adaptation and, given that children all over the world now are very familiar with digital 
interfaces from videogames, the fact that they will no longer need to learn to type means that 
they will find it easy to produce texts in these environments. Of course, projecting ourselves 
further into the future, we can look forward to “telepathic dictation” in which our thoughts, 
perhaps ever pre-verbalized,could be projected directly as text. 

Whatever, the future holds for text creation in EV, there is a general message here that 
should be considered with each proposed technological advance: are the affordances of 
previous solutions to a problem being reproduced in a new technology simply because they 
are familiar or can they be dispensed with? We must remember that the future is not simply 
be a continuation of the past; the passage of time brings with it also a reconfiguration of the 
world that may involve a disruption of what we have come to know and depend upon. The 
future is really a new world and the technologies that will be built to serve us in that new 
world may require us to reconsider practices and devices we once found natural and chart a 


very different way forward. 
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Information Storage Evolution: How Things Have 
Changed and the Potential for Data Loss 


John Bowditch 


I have had a longtime interest in the "permanence" of the historical record and "A Short Note 
on Writing Letters" gave me an opportunity to reflect on the relationship between the 
technologies used to create and thereby preserve the written record and our understanding of 
history. 

The first ways that information was written down involved the painting or carving on 
rocks and statues, etc. These methods assured an extremely durable way to preserve the data 
(think of the expression "cast in stone") but of necessity profoundly limited how much data 
could be recorded. From that time forward as outlined in your piece, the ways to record data 
evolved through a succession of technologies that allowed for ever increasing recording of 
data. These have evolved up to the present time. With each advance, first clay tablets and 
then papyrus, followed by paper, more and more data could be recorded and preserved. But 
there was a price: each new medium certainly allowed for far more storage of data but each 
was more and more fragile. Papyrus and paper can be easily transported but they also can be 
lost due to fire or flood. Think of the burning of the library at Alexandria. Each new 
technology was ever more ephemeral. 

The invention of the printing press (surely the greatest modern invention) hugely 
increased the ability to counteract this trend by making many copies of any data. However, 
the use of acid paper, which began in the 19th century, and which made cheap newspapers 
possible is highly fugitive. Various types of copying machines helped; with James Watt's 
portable machine in 1780, followed by Edison's electric pen and mimeograph as well as 
Bell's wax cylinder "Dictaphone" which evolved in the 1890s. This, combined with 
lithography and the half-tone photo process, introduced in 1880, allowed the flowering of 
illustrated trade literature and glossy magazines with colored illustrations. 

The advent of sound and film recording as well as descendant processes such as digital 
recordings have all allowed for ever greater storage of all sorts of information. But this was at 
a price: information storage devices of these types require machinery to retrieve the data. 
This is problematic as the machinery becomes obsolete, the ability to retrieve the data 
becomes increasingly difficult and expensive. Try to find a means to open a file on a floppy 
disc! Another example would be it is hard to play back a 5-inch concert cylinder (popular in 


about 1900) without the correct type of cylinder player! 
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And when we consider how the advent of online digital storage has vastly expanded our 
ability to store and rapidly retrieve data, we must keep in mind that these digital means of 
storage are extremely fragile: for example, now that indexes like Thomas Register are all 
"online" there are little incentives to preserve data related to obsolete items which formally 
was preserved in old catalogs, etc. Also, now that CDs and DVDs are rapidly being replaced 
by streaming services, controlled by large corporations, multiple copies cease to exist. An 
earlier example of this trend was the destruction of a huge percentage of early motion 
pictures as the studios saw no financial reason to keep them. Fortunately, a few copies 


survived in various theatres around the world. Otherwise they would all be gone. 


An extreme example would be a major war or other disaster that seriously interrupted 
or permanently destroyed access to these materials. This would mean wholesale loss, a 
modern and even more devastating equivalent of the burning of the library at Alexandria with 
its concomitant loss of the knowledge of Classical Antiquity, an event which set back Western 


Civilization for a thousand years. 
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URL as Space Time Encoder 


Leon Van Kammen 


A continuation of Soace Time Compression 


I've observed something that many humans (including myself) appear to enjoy: 
We tend to (un)compress spacetime constantly. 
Right now, I’m typing these characters, to compress my thoughts over time, into text. 


At the same time you, the reader, seem to decompress these peculiar symbols back into 
thoughts. 

The logistics for each character are dazzling: keypresses, cables, electricity, pixels, bits and 
bytes, ink, network requests, speech, sign language and so on. 


Compressing space and time is something we do every day. For example, when you 
share a URL (or, in popular terms: a cat video) with someone you care about. 
The textual link represents a compressed version of the experience, condensed into abrief 
sequence of characters. 
We tend to not to think about this much: but URLs just work. 


URLs, the invisible app, gives us the freedom to address something without having to 
degrade to a "let me tell you all about it"-session. 


URLs are the textual vehicle which carry us to a symbolic mosaic elsewhere. 


This makes one think: perhaps the invention of the URL itself was the core, and the rest 
(HTML) was just icing on the cake. 


The power of URLs is not easy to realize for most people. 
Take for example the marketing slogan: 'Come for the app, stay for the network." 


Well, hypermedia URLs are actually somewhere in between; they are very invisible, yet 


essential. 


When we want to compress spatial thoughts, we tend to use descriptive phrases like 'it 
looks like this and that' (‘let me tell you all about it’). 


If we want to compress this further, we might start drawing something in 2D or 3D. After all 
that hard work, the moment we want to share something electrically, that's where the URL 


comes into play (in one way or another). 


These URLs, these compressed textual ‘synapses’ make up the web, just like synapses 


connect things in our brains. 


In the following chapter, let’s dive a bit deeper into hypermedia and URLs. 
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Hypermediatic browsers 


In the last years, I seem to have slowly departed from my original understandings of 


hypermedia experiences: 


The carrier is the URL, the modulator is hypertext (HTML) 
HTML is then (later) modulated by other (hyper)media (images, sound, video). 


It’s right in front of your eyes - 


Let me first start by saying that I’m very grateful for HTML & web browsers, it has brought 


us so much. 
Second, my introduction to Bulletin Board Systems (BBS) & internet was cultural, as in: 
it presented itself to me through friends swapping floppies, phone numbers and webring- 


URLs in the classroom. 
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This means the ideas below, should be viewed through the lens of somebody who got 
introduced to personal computing through personal computers, not books or pioneers. 

Back then, HTML was presented to me, as the BBS’ es of the future, bundling all the 
telephone numbers into one telephone number (ISP). 


However, these days I can’t help noticing that the persistent HTML-first focus of most web 
browsers is perhaps an indication that it’s just a possible version of the web. Having become 
familiar with XR (AR/VR) on a day-to-day basis, the following hypermediatic vision for ‘ the 


web’ started to make more sense for me: 
The carrier is (actually) the URL, the modulators are hypermedia (URLs) 


File Formats can be modulated by themselves again by embedded or referenced text, 


HTML, audio, animation, and URLs to allow interesting feedback loops e.g. 
OK, now we’re entering slippery territory. 


Let's go all in, and make it even more slippery by adding some McLuhan sauce: 


The URL is the message 
HTML is JUST a (hyper)media format 
The browser is JUST an app. 


Let's dive deeper into these ideas, which are currently drafted as a spec at 
github.com/coderofsalvation/hypermediatic: 

If we want the web to be an inclusive public space for spatial computing or games, 
maybe we need to move beyond a 'text document'-web with bells and whistles. 

We could embrace a hypermediatic web, and actually start hyperlinking all the things, and 
mediate things on a browser- or OS-level, not on an embedded scripting-level (javascript). 

For example, integrating ideas like Hyperwords>? ought to be a no-brainer for any 
operating system vendor, to enable textual hyper features, which can foster hyperlearning, 
and hyper economies. 

Through the lens of a hypermediatist, we’ve only scratched the surface of hyper 
connecting things. 

The alternative is, as I will elaborate later, that all native high-performance experiences 
will gravitate toward non-hyperconnected native apps within proprietary app stores. Basically 
app stores as an endpoint, preventing many to many connections (hyper). 

Instead of talking about the so-called necessity of non-hyper networks (scarcity/profits), 


I will focus on the potentials which might hide beyond these limits. 
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Even for hyper capitalists, postponing the profit-necessities could introduce interesting 
benefits compared to their competitors (just look at how Amazon has postponed profits for 
years). 

Now, I kindly request that you focus on the spirit of the following ideas rather than 
getting caught up in the forthcoming conflation of 'hyper' terms. The reframing of 
hypermedia towards hypermediatic raises several questions that challenge current 
hypermedia: 


1. Until which point is HTML holding back high performant 
hypermedia-surfing of images/sound/video/3D? 

2. Why are web browsers poorly supporting other file formats & protocols 
using URLs? And when should the individual (not) decide which file 
formats / protocols are deferred to the browser-extension backseat? 

3. Why are hypermedia file formats not encouraged to support the 
outbound/embedded hyperlink-paradigm like HTML? 

4. Why are operating systems not supporting/encouraging 
outbound/embedded links in file formats out of the box? (liquid information/hyperwords 

e.g.) 

5. Is a scripting language, embedded into documents an anti-pattern for 


hyperlinking / hypereconomy / hypernavigation? 


A lot of it, can perhaps be explained by the hypertext-first-hypermedia-later vision which 
most web browsers and operating systems seem to share. 
From an historical perspective this makes sense (it was easier to project text than images back 
in the days). 

However, it also means that whatever file format is consumed in the web browser, gets 
served through this lens, through HTML, or indirectly via browser extensions. 


This however, makes hassle free native hypermedia experiences a second-class citizen, 


because the web browser doesn’t seem to motivate the user: 


¢ To browse to an URL inside an image (when surfing to foo.com/bar.png e.g.) 
* To browse to an URL inside a video (when surfing to foo.com/bar.mp4 e.g.) 
¢ To browse to an URL inside audio (when surfing to foo.com/bar.mp3 e.g.) 


* To switch mediatype (switch from surfing audiolinks to videolinks e.g.) 
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These days, all media file formats have metadata support for URLs. 

Actually using them would allow surfing video-only, audio-only and images-only, basically 
author-curated webring-like explorations without the privacy/security burdens of 
hyperscripted HTML hypertext. 

We are not departing from text here, rather the opposite: we are liberating text. 

Why not liberate hyperlinks from it’s initial messenger (HTML), and allow hypermediatic 
hyperconnectivity across any messenger/fileformat. 

This perspective turns URLs into watertubes which connects all our houses, not caring 
whether the water is used for making tea (HTML), coffee (PDF), or beer (3D file formats 
e.g.). 

Certain browsers already do this, by supporting outbound links from a PDF to a website, and 
vice versa. 

In the same sense, we can link to VR/AR experiences inside a PDF. 

Hypermediatic browsers don’t treat HTML as some sort of doorway for hypermedia 
experiences, because this doorway means future hardware (file formats) are treated like 
refugees right off the bat. 

Perhaps traditional web browsers could become more hypermediatic by introducing a 
filetype- or protocol-store (besides a browser extension-store), to foster interesting 
hyperlinked hypermedia experiences with native performance (skipping the ever-growing 
HTML-stack for hypermedia experiences). 

Hypothetically, when the user enters an unknown filetype in the web browser URL-bar, 
it could trigger (mediate) a search in the filetype- protocol- or browser-extension-store (and 
let the user install it or not). 

The argument of HTML web browsers acting as a safeguard, to protect the users seems 
contradicting because it goes against decentralized technology (the users should decide, not 
an external safeguard which will paradoxically increase its own centralisation). 

To further confuse you: I love the W3C and all its amazing protocols, they are an essential 
organism of the web. 

In this sense, the URL becomes the carrier, not HTML, and the browser becomes 
hypermediatic in dealing with file formats. 

At some point, perhaps we have to conclude that a lot of innovation (native apps, blockchain 


e.g.) seems to still happen through- or on top of URLs. 
The URL is hard to escape. 
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Examples of hypermediatic browsing: 


1. A share-button, is a self-hypermediatic feature,which modulates an URL with an URL. The 


moment we press a share-button, a URL is being shared somewhere to something. 


It calls another URL, to share a URL. 


2. A GPS URL: 
://somefile.extension#q=<latitude>,<longitude> 


It is very interesting why our browsers are not motivated enough to standardize URLs 
towards spatial ‘globe’ browsing. This could encourage people on a local level to annotate/ 
capture/map their environment, and add hyperlinks to other topics/places/photo URLs etc. 
We don’t need ALL gps metadata at our fingertips, just a globe. 


On this globe, surfed documents would just appear on certain coordinates on the go. 


3. A 4D URL: 
://somefile.extension#pos=0,0,0&rot=0,0,0&t=1,0,100 


Xrfragments.org is motivating browsers to adopt spatial browsing. 


The URL qualifies as a the textual bad guy in a James Bond movie, who never dies and 
always keeps coming back. 

Perhaps we’ve got so much focused on hypertext, that it made us blind to the idea of it just 
being a messenger of hypermedia (not the other way around) 

Perhaps this realization, might also explain why lots of developers and technology 
corporation’s seem to have wandered off into hyperscripted websites, the opposite of 
hyperconnected hypermedia. 

Right now, a developer seems to need thousands of lines of code to send a ‘hello world’ to the 
screen using the latest native/web app technology, with or without expensive AI 
infrastructure. 

These are perhaps defining signals, to indicate we’re somehow reaching a ‘mainframe’ - 
phase again, like in the 60s. 

These ‘signals’ more or less pushed me towards creating the XR Fragments*+ hypermediatic 
specification & related demos. 


Fun fact: after going thru the whole adventure of creating the spec & demos, I read this 
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line in the Future Of Text (vol3) afterwards: 


“Imagine three-dimensional documents, leveraging advances in spatial computing (AR/ 
VR), in which following a hyperlink brings you to a new location, but retains its relationship 
to all other locations.” ~ Adam Kampf (A brief history of cortex) 

This is exactly the premise of XR Fragments, with some icing on the cake: it can be 
applied to all major pre-existing three-dimensional file formats, and allows interlinking 
remote/local text as well. 

Looking at hypermedia through a spatial hypermedia lens, not a hypertext lens. The 
idea of adding an inverse relationship to the mix, between hypermedia and hypertext, allows 
for a more inclusive hyperlinked vision of hypermedia browsers: 

A hypermedia browser should ideally treat hypertext (or other file formats) as a 
modulator AND/OR carrier. 

The same evolution happened to music synthesizers, when FM synthesis entered the 
stage and liberated itself from the fixed carrier/modulator setup of the first synthesizers. 

Suddenly, carrier/modulator combinations could be freely selected, allowing an 


unlimited variety of sound colors. 

In the same sense, the fixed HTTP/HTML-combi is one of the first hyper synthesizers. It is 
time to liberate ourselves fully, and unlock the full potential of protocol-agnostic & file- 
format agnostic synthesizers. 

Perhaps the quest of the hypermedia browser, or organism, is to pat the HTTP/HTML- 
combi on the back, put it into an appropriate corner, and promote equality across all past- and 
future file formats & protocols. 

In the following chapter, we’re going to imagine how we could pat LLMs on the back, 


in a similar way, and put them into its appropriate corner as well. 


LLM cake or icing? 
The human experience seems (for most people) to be a quest to improve things around 
yourself without bothering other people too much. We evaluate this not based on deduction, 
like Large Language Models (LLMs), but rather a peculiar combination of induction, 
deduction, abduction, fear, caffeine, overoptimism, (lack of) sleep, and many other things. 
With this cognitive monkey cocktail, humanity has achieved a lot of things. 
LLMs are reductionist historicist pets, they are like commander Data (StarTrek Next 


Generation) who loves to solve Sherlock Holmes mysteries (after reading all Sherlock 
Holmes books). 
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The moment he has to solve a mystery of a book he didn’t read, things get hard. 


It is very hard for an LLM to ‘be’ like Sherlock Holmes, but surprisingly easy to talk or 
sound like Sherlock Holmes. 


Besides the limitations, the LLMs are fantastic assets in the ‘automation’ arena. Automation 


means batch: we know what to expect. 

Perhaps personified LLMs (chatbots) are an opportunity to show children that LLMs 
are nice story tellers and great at gaslighting, but that truth is rather an activity, which 
happens in the human minds (by verifying things using induction, deduction and abduction). 
The funny LLM-friends, only use deduction. 


It might also help us steer away from our historicist urge to predict the future based on 
the past (rather than considering that the future emerges from multiple things, in the spirit of 
Karl Popper). 


We have this historicist urge, because we love simple explanations. 


We love to read an Forbes-article, look at a timeline and say, "Forbes told me that company X 
was the first to do Y." 


Reality, however, always turns out to be much weirder and more complex than we think it is. 


Interestingly enough, we'll always look back on how clunky things were in the past. 
Right now, we’re in a similar phase with LLMs. 


We’re already starting to look back on 'clunky' AI and chatbots as just the beginning. Perhaps 
it's only a matter of time before LLMs are the norm in computing and user-interfaces, but 


more on the backseat. 


The exciting parts will not disappear: LLMs make computer accessibility so much easier. It's 


so much easier to understand a computer with the help of LLMs. 
They can definitely make it easier to understand code, the command line, and documentation. 
LLMs are definitely useful for preventing otherwise cognitive overload. 

I love LLMs to be the stick or filter our monkeybrains needs, to improve ourselves. 


But when LLMs are presented as the cake, making us the icing (personified chatbots who 


lecture us), then it feels a bit like riding a horse backwards. 


The opposite seems much more attractive: beautiful efficient user interfaces which our 


monkeybrain and muscle-memory can operate on, with LLMs being the icing on the cake. 
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Connecting Knowledge in Space 


Frode Hegland 


A key opportunity of XR will be displaying and interacting with large knowledge 
connections/maps/graphs. Key aspects will be efficient ways to choose what is shown and 


what is hidden as well as how to connect graphs. 


Show, hide & select. Using the Map view in Author has been considerably easier now that 
the user can assign tags to nodes, such as ‘person’, and through a keyboard shortcut hide all 
nodes tagged as persons, and then when showing them again they are all selected. This means 
that the user can easily select based on tag across the Map and perform layout functions, such 
as simply turning them into a column (for example). To make the hiding and selecting more 


transparent to the user we plan to implement a ctrl-click menu in Maps, which is coming up. 


Related is the issue of nesting or connecting Maps. What we are thinking about is 
something as simple as being able to either drag an Author document onto the Map and have 
it appear as the document title, or name if no title has been assigned yet, with [hard brackets] 
to indicate it’s a link. Click and it will act as any link in Author with a ‘Delete’ & ‘Jump’ 


option. 


Another issue is how to embed the Map in the PDF. Upon export from Author to PDF, the 
Defined Concepts (the nodes on the Map) are exported as Glossary terms. We are looking at 


how their relationships can be encoded in Visual-Meta. 


XR experience. In Author we have the Map view which Brandel have put together a test 
version of in XR. 
https://thefutureoftext.org/2022/09/19/vr/ 
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The Unintended Long Shadow of Print in The 
Twenty-First Century 


Mark Anderson 


This article broadly relates to PDF, and the unintended damage it has done, but such is the 
polarity around feelings for the format, it seemed divisive to put in the title. I love print and 
books, so I might seem an odd author for this piece. But practicality and expediency has been 
a poor friend to our future selves. We owe ourselves better. 

Back in the early 1990s, before the Web had achieved notice or scale, a different digital 
problem faced us. How to print (to paper) a document on one personal computer (‘PC’) a file 
created on a different personal computer (regardless of OS: Microsoft, Apple, etc.). We easily 
forget the erstwhile difficulty of this task. 

1992 saw he advent of Adobe Acrobat and Postscript-based documents. Essentially, the 
text of a document was converted to vector shapes drawn on a virtual sheet of paper, saved as 
a ‘Portable Document Format’ file or PDF. THere were additional options, but this was the 
key advance of Acrobat documents. If I made a PDF and sent it to you (with Acrobat-capable 
software) it printed out just as I intended. Mirabile dictu! It was a big deal at the time, you 
had to be there. 

So all went well then? Not quite, but not through intent but a combination of 
expediency and indifference (‘not my problem’). The issue is not printing (and who prints to 
paper these days?) but document fidelity. Step forward to the late 90s. A new phenomenon, 
the Web had arrived. Write once, publish once. Print once? Well, not quite. I I sent you a web 
(HTML) document and forget to also sent any images/scripts/stylesheets, your ability to see 
my document as intended was lost. But, PDF to the rescue. ‘Print’ my page to PDF and in 
one file I can send the intended render to anyone with confidence they will see my render of 
the content. Note the ‘render’, already the paper element is not central though at the time we 
are all still printing documents to read them; screen-reading had yet to take hold. 

Another factor in this story was the US bureaucracy’s adoption, especially by the US 
judiciary, of PDF as proxy paper of in the late 90s. Where such large players moved, others 
followed. A digital document (replacing the need for) a paper document was non-trivial, the 
paper copy of record was not longer an absolute requirement. We also cannot overlook 
capitalist commercial advantage of monopoly control of key formats. 

However, if fewer trees are felled for paper is not that a win for us all? Absolutely, but 


what the non-tech community missed in this is that putting text into PDF is not a 
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transparently two-way process. Text that goes in may not come back undamaged. Why so? 
Consider the original task: consistent print output. It needs to look correct on paper (and on 
screen). But, unseen in the way the process is effected, the source text is typeset in a no- 
reversible way. The for-print process simply does not care how the fidelity to paper-print is 
achieved: it just has to look right. 

As a first example, old, pre-digital, type has evolved visual affordances like 
ligatures.This gives an aesthetically pleasing result. An obvious example is where the 
successive letters ‘f’ and ‘1’ are replaced by a special ‘fi’ type character where the dot of the 
‘ie’ and top end of the ‘f are conjoined; the same for ‘ff’, ‘ct’, ‘at’, sequences, etc. Ironically, 
I cannot produce those ligatures here in this authoring tool. Ligatures are fine for print, but 
overlooked is that most digital typefaces (fonts) used for on-screen text rendering lack 
ligature glyphs or the means to decompose such back to their constituent letter glyphs. So 
into the document goes ‘office’, whilst out comes ‘oce’ (if there is no ‘ffi’ ligature glyph in 
the font used to render the output). Perhaps not quite what we would have planned, had we 
thought ahead. But recall, this was not the goal of the original PDF. 

A skilled typesetter avoids ‘rivers’—unintended gaps in typesetting which appear to run 
through a paragraph of text due to a coincidental alignment of spaces. In an age of automated 
type, this leads to ‘soft-hyphenation’ where words may be broken—if the need arises—across 
a line end in order to assist with spacing of characters in type, especially when using justified 
(i.e. non-ragged margin) text. If the output is only print, then there is no need to record the 
presence of a soft hyphen. But, copy-paste text from PDF to another digital document and 
the break of ‘outward’ across a line end becomes ‘out- ward’. Why? PDF simply has no 
(standard) way to undo the typesetting damage to the original source text, nor does simply 
removing all instances of ‘- ‘ help as this damages other legitimate text sequences. 

Another issue is that a to-typeset process simply needs to typeset the (virtual) page. At 
minimum, most printed pages will contain a page number, if not headers, footer sand other 
legitimate print page elements. But if I select text in a PDF across a page break, do I want 
(expect!) the page number header/footer, etc., to turn up in the copied text? Likely not, yet all 
that unwanted content is dumped into the copied text (N.B.: in some cases the PDF and/or the 
viewing software used may have figured a way around this). 

It is easy to point the finger at somebody, anybody, else for the failure to see and 
address this, but we are all complicit. By allowing poor standards to pass by us without 
comment, we accept them. Had anyone been bothered, the PDF specification could likely 
address all the above failures, but it does not. Whilst new changes would only feed into new 
tools and likely could not fix existing document, it would be a start. But inaction is rarely a 


route to improvement. Meanwhile, we keep loading use of PDFs way past their intended task. 
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It is an irony that an early benefit of PDF use was ‘printing’ web (HTML) pages such 
that the result was a single computer file with a consistent render. Ironic because in Future of 
Text discussions it is apparent that when shifting text usefully between the normal digital and 
XR spaces, a format with addressable document structure is a real benefit. Indeed the HTML 
document with explicit/implicit outline (DOM—Document Object Model) as well as styling 
separated off into CSS (Cascading Style sheets) turns out to be one of the best options. 

So can PDFs be read/exported into clean, styled, HTML? Sadly, no. Look at the HTML 
and it is essentially a facsimile of the source slugs of typeset PostScript. Whilst that has 
provenance, it is not useful provenance whilst also robbing the exported text of useful 
structure. 

So, what of the PDF? Its strengths remain its strengths and the ability to hold an 
immutable render of a source text is valuable—think copies of record for contacts, treaties, 
medical records, etc. However, it is fair to ask why we persist in printing [sic] to PDF 
documents that still need some flexibility for remediation. 

Our future selves will not thank us. Nor can we idly blame “somebody else’. There is 
no future but what we make. So it is time to start re-visiting our notions of what we mean—or 
intend—when selecting File > Print, not least in light of the advent of the new possibilities of 
XR. We have a generation or more of semi-re-usable PDF documents but we do not need to 
keep making past errors. Print-typeset (fixed render) output needs to be re-imagined as a but 
one of a number of possible output forms, and perhaps not the default, and to be used only 


where immutable display/render is a necessity. 
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‘highchair’ 


Frode Hegland 


highchair, 
high—chair 


This two line poem, composed on one word or two, depending on use, highlighted to me how 
a world experience can be revealed in a basic pronunciation. When referring to a baby 
‘highchair’ most parents would pronounce it as a single word: ‘highchair’. However, 
someone who does not have children or is not around people with children might very well 
pronounce it ‘high-chair’, with a break in the middle. I notice that the American way to write 


it is as two words and British as one word. I wonder at the pronunciation though. 


Do Americans pronounce it as one or two words? I asked ChatGPT: “In American English, 
"highchair" is typically pronounced as /haitfer/. The emphasis is often placed on the first 
syllable, and the "i" in "high" is pronounced like the "1" in "pie." The "ch" is pronounced as in 
"chair." Overall, it sounds like "HIGH-chair," with the stress on the first syllable. Keep in 
mind that there can be regional variations in pronunciation, but this is the standard American 


pronunciation.” 


If you are a parent, you will pronounce this one way. 


If you are not, another. I find this to be profound, and it is something not in the text, it is in 
the pronunciation of the text, the performance of the text. Can we—should we—embed more 


in text? 
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Why XR? 


Frode Hegland 


This week I saw two friends who are not in my industry and whom I don’t get an opportunity 
to talk to about the work we are doing very often, and they both asked about the benefit of 
putting on a headset. The questions were earnest and they simply could not see why anyone 
would want to put a ‘bucket’ on their heads. These two friends are intelligent and thoughtful 


so I owe it to them to answer as clearly as I can. This is for you Elisabeth and Asle. 


Caveats & Context 


I first pointed to the Apple Watch we all wore and—as an illustration of how technology 
moves fast—told them that the Watch has roughly the same processing capacity as a high end 
PC 5 years ago. I followed this up with the expectation that in maybe 10 years an AR headset 
could be as light as a regular pair of glasses today. Imagine a double tap on regular glasses 
and you have a full AR view of your work, I said. 

I then made it clear that I don’t expect anyone to work in full VR/AR any time soon, 
though of course when the devices are lightweight enough, like the glasses mentioned above, 
or when they have shrunk to contact lenses, then maybe, but that is still a ways off. At this 
point I think of them as ‘thinking caps’. 

Similarly I pointed out that they already have made part of the point themselves by not 
doing their main work on their iPhones: they use computers with large screens so that they 


can see more. This is of course a basic benefit of XR: More space. 


Headset On 


Once these caveats and long term perspectives were out of the way, I could start with the 
most basic use of VR which is to put yourself in a different environment from the one you are 
physically in. If you work in a crowded office or dorm, you can put on a headset and visually 
‘be’ somewhere else. This can be as basic as reading a book in traditional form while sitting 


in a nice country cabin in front of a fireplace. Or in space, orbiting the Earth. 


I find it useful to continue with the “reading of a document” as an example, particularly 
in a book called The Future of Text. I suggested to my friends that they may be reading an 
academic paper in AR—so they would remain in their environment—and instead of a single 
or two page view, they could view the document as an accordion folding out to show all the 


pages in a sequence, or perhaps as a large mosaic so that they could quickly move around the 
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virtual ‘pages’. They could perhaps gesture to have all the images in the document move 
across to a physical wall in their room and the reference section could remain in view 
somewhere else. Citations could be instantly checked with a click, bringing forth the source 


document. 


Interlude: Open & Rich Data & Metadata 


These interactions are of course something which can be done in a traditional digital 
document (some of which we have done here with Reader augmented with Visual-Meta) and 
this is something which highlights the importance of the system having access to the data and 
metadata to enable. It’s as simple as this: A system can only let you interact with something 
to the degree of which it understands what it is. If it understands something with semantic 
meaning that provides a much richer opportunity or interaction than if something is 


understood simply as a rectangle. 


Dimensions of Thought 
Beyond the flat document in XR it starts to get more interesting. 


We can continue with a flat, traditional document as the base for a little longer however, 
and we can do some dreaming together. Suppose you are the user and you have been reading 
a document which you feel is a bit dense so you’d like a more expansive view so you put on 
your headset. Now, if the system is well built and open, you should be able to expect to have 
your document come with you into the XR view, but this will not be as possible if different 
companies own proprietary environments and the interconnections are badly thought out. 


Refer to the ‘Interlude’ above. 


Let’s start instead with a scenario where you choose not to take the document you had 
on your laptop into XR. Let’s look at how you find/see/locate open a document instead of 
looking at how you read the document. 

What might this process be like and what might it look like? This is where we can see 
questions opening up. 

Your ‘Library’, to call it that, in XR, can be a 3D representation of a traditional, 
physical library, with extra features. It can be a list, just like you have on your desktop 
computer. It can be a graph with nodes for documents, themes, authors or institutions. It can 
be a landscape, it can be a timeline. The options are vast at this point in time, because we 


have not built and therefore not decided what it ‘truly’ is. 
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‘Real’ Library+ 

Imagine also looking at your physical shelf and since your headset has scanned all the book’s 
spines, you can ask the headset to show you where a specific book is, which will then be 
highlighted, or you can ask it to search inside the books and show you relationships. The 


‘pass-through’ video which allows you to view the world can also analyse your world. 


A Second Interlude: Ownership 


If our documents (primarily for the sake of the scenario presented here, containing traditional 
academic information, with or without augmentations, and other media) are truly ‘data 
owned’ by us, the end user, we will be able to use any Library view we want, from any 
vendor, for any aspect of how we want to experience the data. If data formats become 
proprietary, we will have to use whatever the initial vendors choose to give us, and then make 


areal effort to try to export to use another. 


If the formats are free and open, the user owns the data and can choose what software 


system to interact with it. 


Library of the Mind 


Let’s return to the Library and assume that economic and political battles have been won and 


data and metadata is open, robust, rich and owned by you, the end user. 
What can your ‘Library of the Mind’ be like? 


One thing you can expect is to be able to instantly change what it is, with a gesture, 
voice command or other. You can expect to be able to have the most traditional form of 
document representation and you can expect AI powered analysis and views. You can expect 
wild and crazy things like holding up two documents and telling your AI system to let them 
‘fight it out’ in the XR environment. You can imagine relevant quotes from both documents 
being compared and references being analysed to see if they truly back up the assertions 


made. 


Further beyond this we can imagine learning new ways to visually represent data and 


having the ability to change representations at will. 


& so much more. 
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Walls as Views & Connectors 


At some point we will be comfortable enough in XR to have headsets on for longer sessions 
and this is where putting information pieces around your space can become useful. Meta calls 
this ‘Augments’, which is an interesting term. 

For example imagine your calendar on your physical wall being instantly able to toggle 
from a basic list of events to showing dynamically the weather in days ahead, including 
weather in other locations where you are planing to travel to. Imagine further that days gone 
past becomes easy access to pictures you have taken as well as transcripts from meetings. All 
of this clear and explicit to you, but hidden from visitors to your physical space and only 


showing what you would like it to show to visitors in your virtual space. 


Payback in 2D 


If you are thinking that much of this can be done in 2D on traditional computer screens, you 
are right of course, but innovation has slowed to a crawl for traditional computer systems 
partly because users no longer see them as spaces for innovation so therefore there is little 


demand. 


One of the real benefits of investing in ‘dreaming in XR’ can be bring is exactly this; 


similar innovations—when they can be implemented—in traditional systems. 


XR brings the hype we need to dream again. 


3D expansions 


Our Stone Age ancestors did not simply paint animals on cave walls. They drew symbols, 
most of which we are still trying to decipher. They also used the medium of rough cave walls 
to bring out movement when living flame from torches illuminated them. Humanity has been 
experimenting with eternal representations of thought for longer than we consider ourselves 


to be Homo Sapiens. And we have a long way to go». 


I predict that text will remain a central medium for communication for a very, very long 
time, but it will become augmented with not only moving and still images and interactions, 
familiar in games and on the Web, but also in entirely new ways. 

Brandel Zachernuk created a ‘magic carpet’ of records from conversations. There is no 
text there when experiencing the system at scale, but text can be brought forth to present 
meaning and records. It’s a thoughtful exploration of how text can be presented at different 
scales. 

Text, in the form of alphabetic script, has a tremendous range in expressiveness of 


thought and grammar connected in powerfully augmentative ways. 
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Text is also surprisingly non-visual, despite being a predominantly visual medium, 
giving other means of representation an opportunity to shine. Large amounts of text can be 
summarised into smaller amounts of text but they still need to be read, which takes effort and 
is therefore not glanceable. 

Will the 3D nature of XR with near-infinite potential for re-presentation be able to 
augment text, without dumbing down, without resorting to only a cartoon presentation of 
knowledge? I hope so, I think so, but we will only find out if we dream about work in 


extended reality. Together. 
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Building in XR 


Frode Hegland 


Notes for a grant application made at the close of 2023 to build a system to experiment to 
experience, with Future Text Lab comments below, as a reflection of our effort to plan to be 
practical and realistic and dialog to stretch this all over the place. This was posted online and 


may have been updated later at: 


thefutureoftext.org/year 1-implementation/ 


Year 1 Implementation 


The project aims to demonstrate that working in XR with documents can be useful. The target 
user is any scholar; university student, undergrad, postgrad or independent scholar reading an 
academic paper. The primary interaction for the first year will thus be reading a single 
document from their own Library/collection—not generic text—when seated at their normal 
physical desk and not standing in a specific VR/AR location—in support of this aim. A basic 
synthetic environment will be provided for the user to have a visually calm environment to 
work in and the user will be able to toggle between AR and VR. The initial work will not be 


focused on this background environment but on the reading experience. 


PLEASE NOTE: This is all only an initial direction to help us think and build. Actual 
implementations will likely change as we learn. The core functionality will be for the user to 


be able to read their own academic documents in XR as usefully as possible. 


Library 


Access. When launching ‘Reader XR’ (as an app or webpage, still to be determined) the user 
will be presented with their Library of documents, after an initial setup for the user to inform 
the system how to locate the documents they use on their desktop/laptop computer. This 
transition in and out of working in a headset will need be as smooth and quick as possible. 
Initial XR View. The initial view of the Library will be showing the most recent documents 
the user has interacted with in their traditional computer environments at ‘the top’ of a list, 
serving the premise that they will, at least for this research phase, already have chosen a 
document to view in XR and it will be one they just viewed before entering. The very first 
view of the Library will probably simply be a list of documents by title and author name (if 


we can parse Visual-Meta in XR at this point) or document name. 
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Send from Desktop to XR. We will also research and test methods for the user’s traditional 
PDF viewing software to ‘Open’/Send a document from the desktop software to XR for more 
instant access, skipping the Library. We will provide and document our own connection from 


Reader for macOS as a test case for developers to use. 


Document Interactions 


Initial View. When choosing to read a document, the document will initially appear as 
though floating in front of the user, just a little over their (physical) table. The user can 
change this default view easily. 

Document Interactions. The user will be able to directly interact with the document to move 
it by grabbing and dragging it with one hand or turning it and scaling it with two hands. 
Document Views. The user will be able to read as a single page, two page spread, multiple- 
page spread or as full document with every page displayed wall-size. 

Physical Desk Use. Interacting with a document floating in space is not ideal, so for the user 
to annotate or interact with specific text, the user will be able to point to a page and the page 
will instantly move to the surface of the user’s desk, where they will then have the physicality 
of the desk as their substrate when they point to the document and interact with it to 
highlight, annotate and look up selected text. (Such interactions will include highlighting text, 
if we are able for the XR environment to render full text. We may end up displaying pages as 
images for other views and as glyphs when on the physical desk.) 

Component interactions. The user will be able to interact with the document to put elements 
from the document in spatial positions either manually or to pre-determined locations, 
including images, table of contents, glossary, graphs and references. (achievability and final 
form depends on research deliverable of parsing Visual-Meta in XR) 

Connection interactions. The user will be able to interact with citations in one document 
and explore how they connect to other documents in their Library and beyond at some point. 
(final form depends on implementation-research of implementing views of connections, an 
exciting part of the work). 

Future/Advanced Interactions. Interactions with knowledge graphs will be investigated, 
involve questions of how document knowledge graphs connect or interact with user’s 
knowledge graphs, how to hide and show nodes, how to nest graphs and more—using the 
extended space without ending up with overwhelming clutter. 

Research will also be undertaken on how to view connections between documents as 


references and links; how to view them and control them. 
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Email to the team 
Morning, 
* Is this a reasonable list of actions, in that it would be useful? 
* Is this a reasonable list of actions, in that it would be achievable? 


¢ Any other thoughts? 


Replies with general comments 


Replies for Leon Van Kammen and Mark Anderson for clarifications were agreed and 


implemented. 


Reply from Alan Laidlaw 
(As a former product manager,) I think the 2nd section is heavy on unproven implementation 
details. Regardless of whether it is one the right track or not, that level of granularity really 
constrains discovery and adaptation when it is most critical. 
When implementation details are presented like that, I push back. My team would need 
to know why they are critical, how they lead to a desired outcome, what value do they bring. 
It’s hard to figure all that out before the ship has sailed. So instead, trying framing it 
like Ponce De Leon: We intend to sail across the ocean to find the fountain of youth. If we 
don’t, we expect to bring back gold and spices. We intend to spread the gospel and kill 
however many natives please the Lord in order to find any of the aforementioned items. 
Failing that, we intend to map the territory for further expeditions.) 
More Practical Examples: 
H1: XR is a new medium for receiving, comprehending, and structuring text. 
H2: XR will be a preferred medium for managing certain kinds of text-based tasks, ie deep 
work. 
H3: direct manipulation will engage the user’s other senses and enhance or expand their 
palette for interacting with text. 
(In 2D documents, highlighting is an example of direct manipulation/interaction that imprints 
meaning onto static text. We are looking for similar techniques in 3D.) 
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Replies from Alan Laidlaw after changes, based on first email 
That’s a good direction. 

More word salad (Not An Article - just impressions to foster more solid narrative): 

This is an exploration into a new medium, similar to the research that might be 
conducted in a Silicon Valley R&D lab. However, we can explore paths in XR that tech 
companies are inherently disinclined to take. 

For example, when the iPad was released, it trumpeted some ambitious academic 
interactive books like An Inconvenient Truth. But within a year, the innovation fizzled. 
Maybe there’s no money in academia or there just wasn’t a market. 

Direct-manipulation interactive text, as a format, was cut short. 


Lesson: Tech cos are constrained by market forces that limit the definition of desirable. 


They skew toward platform lockin. 


When a nascent medium suddenly becomes available to the public, the solar flare of 
excitement can burn associations in place. Behavioral patterns settle and UX ossifies, often 


way too soon. 

If a medium can evolve gradually, in analog time, it tends to have more range, more 
maturity, and is not centrally owned. (Missing: an example of this. Books? Newspapers? 
Comics? Cinema? Is this ever really the case?) 

Cloud-based software makes scaling new mediums overnight trivially easy, as seen 
with chatGPT. When this happens to an expressive medium, it achieves product status and 
ceases to find out about itself, like someone who gets married right out of high school. It is 
less likely to be a vehicle for art and exploration and more like to be used for entertainment 
and exploitation. 

XR should be allowed more time to grow into itself. 

It’s unlikely that will happen in an industry that values time-to-market above all else. 
(For XR, academia might be a better incubator, as it was with AI. (This is a very debatable 
point.)) 

Anyway. Where was I? 

It doesn’t seem like the tech industry is focused on exploring XR in this way. Or if they 
are, it’1] look like Microsoft Word in 3d. 

Game developers often try to gild a 2D game with XR affordances. But as they gain 
intuition about the inherent material (the grain) of XR, (aka when they get their sea legs) they 
have started to rethink fundamental game mechanics. (Unlike, Microsoft, game developers 


are incentivized to explore novelty.) 
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Early experiments with XR text will hopefully follow a similar course. It'll start with 
XR as a veneer on top of the same old 2d rules of reading. That novelty will fade. If 
innovation doesn’t go any deeper, it’1l probably be abandoned like Apple’s interactive books. 

But if we learn from games, embrace novel interfaces and devices (like behavior trees) 
XR could lead to a broader and more enriched understanding of what “reading” really is. 


What does it mean to “read” a mind? 


Personally, I hope to avoid the use of the word “document” unless it can include things 
that we currently call documents. I think it biases the framing and subsequent solutions to a 


very outdated skeuomorphism. But I would be genuinely happy to be wrong about that. 


Second Reply from Alan Laidlaw 


I'm still on that outside of this, so I only offer opinions from my vantage point. I know how to 
take things from dream to recurring revenue, and I know the messy middle particularly well. 
So to reinforce Mark's points: 

- Stick to abstractions whenever possible, like for the pipeline. 

- Diagrams can save a lot of words. X ~> Y 

- Ruthlessly scope down. 

- Hint, don't promise 

Scope will creep in regardless, and some of it will be necessary. But it's impossible to 
predict right now, so delay for as long as possible. 

If the project is focused on the UX of reading, the library collection is an 
implementation detail. If the focus is on reading text, and “reading” text extends to 
understanding it, that makes the page itself an incredibly complex and intriguing tapestry. 

There is the book; and here is your mind. As you go through the pages, either you 
understand perfectly everything the author has to say or you do not. If you do, you may have 
gained information, but you could not have increased your understanding. - How to Read a 
Book 

Instead of a technical focus, perhaps focus on ramifications. As a primitive, the thread 
(think reddit or slack) and block-level comments improve this email wall of text. But they are 
band-aids, XR could foster much better design primitives. That has applications far beyond 
reading a whitepaper. 

If the board expects some technical proof of work, well that is a pickle. 

(It's a pickle because some the easiest seeming stuff to do in code it usually the hardest. I 


would think an XR state machine diagram would be easy but it could be really hard.) 
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If it were me, I would talk about multiple ways of viewing the material as a hypothesis 
for what XR is uniquely capable of presenting. I would avoid anything that relies on low 
latency or a "real-time" feel. I'm more interested in holding an idea in my hand than a 
facsimile of paper. An idea could be a block of sequential text, a diagram, or a (warning) a 
search query shaped like a man-of-war with a hundred components loosely assembled 
together. (jk) 

To me, the key is to break away from, to unlearn, the page and screen and browser. 


unlearn the rectangle. The trick is do that without filling the vacuum with anarchy. 


I can't stress this enough, though. 70% of what I wrote today should've been in 
footnotes, or tucked under a fold. It didn't need to be read and yet I kept it in anyway. That's 
1) a strong signal and 2) a limitation of the format and our understanding of writing. (editing 


might be a form of gaslighting) 


Reply from Peter Wasilko 
Greetings All, 

I know that we don’t want to over promise, but I am deeply concerned that the proposal 
reviewers won’t see enough "there, there" to differentiate what we have in mind from the 
stock Apple demo video of spawning virtual monitors to show new documents. 

I think we should be talking about providing means of abstraction and combination, 
particularly in terms of view specs and cross-document selection queries. 

As you know from my symposium contributions, I’m a huge advocate to exploring 
linked Benediktine Cyberspaces where a point can be viewed as a document meeting a 
query’s conditions or as query for such documents that can be scrubbed through a data space 
while being simultaneously unfolded into a new linked space dynamically presenting query 
results. 

In short, I think we need to focus on things that can’t be effectively explored in just 2- 
Dz. 

I don’t want to just pluck out and load citations or even create simple typed links 
between them, I want to be able to articulate complex queries like “Show me who has written 
hypertext and/or chi papers that transitively cite JCR Licklider’s Libraries of the Future and 
Doug Engelbart’s “Authorship Provisions in AUGMENT" whose labs or companies are 
hosting GitHub repos that use typescript or javascript — arrayed by citation impact, 
publication count, and lab size colorized by relative funding levels. ” 

Then I want to be able to use a virtual vector-deflection driven cursor to reach into the 


visualization to select specific data points to be popped up as Biography cards on a virtual 
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cork board overlaid in front of an unused wall. 

In short I want an XR Command Line Interface to let me use text to drive what I’m 
seeing; how it relates to the physical desk in my office is but a passing consideration. I might 
want the visualization to be a 3 foot sided cube on my desk, scaled up to a 9 foot cube in the 
middle of the room if several of our avatars are virtually standing around it, or just projected 
as a3 inch cube virtually glued to a physical white clipboard I am holding in my hands. 

Basically, I think we need to show that we are thinking beyond what Apple is doing, 


even if it takes a few rounds of funding to get there. 


Frode Hegland's Reply to Suggestions 


In summary: These are great thoughts but we need a firm foundation for the user to interact 


with their own documents first, then we need to experiment as you have suggested. 


Alan Laidlaw reply to Peter Wasilko 


This: “In short I want an XR Command Line.” 

I think it’s an excellent idea to gather more examples like: I want to be able to... so that 
I can... 

That makes it real, gives it juice. I mean the project, not the proposal. User stories are 
not about promising, but trying to align on what the problem is. 

Anchored to jobs to be done avoids preconceived notions of how it should be. 

Here’s my riff: 

I want to be able to pick up where I left off, except that I don’t want to be surrounded 
with tabs and screens. I’d rather a summary of what I was doing last than to jump back into a 
cyberpatch of unfinished distractions. (I want it to clean up the mess I made.) so perhaps have 
it show me the last thing I manipulated, not just opened or glanced, and where did I focus 
most of my effort in the last session. 

I want the power of command line but one that includes voice and gestures as well as 
text. If it could do all that, I could see going into XR more often. 

Fun exercise! 

On your other points, Peter, you may be right. I have no experience with proposals like 
this. If they want more steak than sizzle, I hope they’d would also want to see research that 


supports the solution. 
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About Data : Metadata 


Frode Hegland 


Visual-Meta is an approach to augmenting documents in order to enable rich interactions in 
a robust manner. This article is the front page of https://visual-meta.info and added here 
for archival purposes so that you, dear reader, may read this instantly without clicking on the 
link today or in the far future when the server is long gone. Visual-Meta is used in this 
document and it enables interactions as described in the ‘Reading this document in 


Reader’. 


The Traditional Book Approach 


A printed book features a printer s page before the main text, with metadata including the 


name of the author, title of the book and imprint information and so on. For example: 


Copyright © 2010 Alberto Manguel. 

All rights reserved. 

Printed in the United States of America. Library of Congress Cataloging- Manguel, Alberto. 
A reader on reading 

ISBN 978-0-300-15982-0 (alk. paper) 


The Visual-Meta Approach 


Visual-Meta puts this metadata, and more, into an Appendix at the back of the 
document instead of at the front (to make it less obtrusive), written out as plain text in the 


very human readable format, based on the academic BibTeX format, as shown below: 


@ {visual-meta-start} 


author = {Manguel, Alberto}, 
title = {A reader on reading}, 
year = {2010}, 

isbn = {978-0-300-15982-0}, 


@ {visual-meta-end} 
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Characteristics 


Enables Powerful Interactions 


* Copy & Paste to Cite. Copy & paste text as full citations, automatically. 


* Click to Access Citation Information. Click on a citation to open the cited document 


directly, not just a download page 
[quick demo: https://visual-meta.info/powerful-citing/ 


¢ Flexible Views. Change the view of the document as you see fit, to see it as an outline, see 
how glossary terms connect, and view citations maps with ease [quick demo: https:// 


visual-meta.info/flexible-views/ 
¢ Glossaries of Defined Concepts for use in Maps and Graphs in software which supports it 


¢* Computational Text. Support advanced interactions, not only static text, for similar to 
what computational notebooks enable, with clearly visible data, including embedding 


executable code 


¢ Connected Text. Makes clear citations for citation analysis and supports link types and 


external resources, such as LLM hints, glossaries etc. 


Rich & Open 


* Rich. Supports a large amount of metadata 
¢ Open. Anyone can read the metadata with no special background or tooling, now & in the 
future 


* Free. No license constraints. Anyone can freely build tools to parse and append Visual- 
Meta 


¢ Extensible. Anyone can add their own fields—not only have predefined fields available— 


should they wish, as long as they make clear what the fields are 


¢ Accessibility focused. By providing useful data and tags for accessibility, (including 
option to include pristine full text for reader software to present in specific ways) all users 
benefit 


¢ Augments XR. Allows for temporary appendices to record spatial information 
¢ Augments AI. Gives AI clear and authoritative information, literally 
* Robust 


* Robust. The metadata cannot easily be lost if the document changes versions or even 


formats since the metadata is not embedded, it is on the same level as the contents 
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¢ Document encapsulated. Does not need a server to be maintained to function 


* HTML Compatible. Can embed full sections of HTML to provide rich interactions to 
readers which supports it 

* Compatible with all valid PDF viewers since it simply adds text to the document. 

¢ Benefits: https://visual-meta.info/legacy-safe/ 

¢ Student End Users — When authoring, copy and paste to cite. When reading, click on a 
citation to see the source and then click on the source to open it directly, as well as having 


much more flexible views of the text than plain PDFs afford 


¢ Educators — Better understanding of student thinking through the Glossaries of Defined 
Concepts, clearer access to citations and simpler environment for students 

¢ Publishers — For production, sync’s up with their XML data for cheap production and 
connects their work better for more use of their content. For ingestion of manuscripts 
makes extraction of metatada faster and places much less onus on the author to format to 


style, which costs effort. Currently in testing with the ACM. 


¢ Software Developers — Easy and cheap to produce Visual-Meta based on the data in the 
manuscript and also easy and cheap to extract from PDF, with no licence or external 


software needed 


¢ Apple — Encourages an open ecosystem where developers can compete by building the 


most powerful user experiences 


All metadata standards can potentially accommodate all metadata but there are real 
differences in how it is produced and to what degree it becomes accessible. Anything can be 
put into metadata fields but at what production cost and at what effort to access it? 

The inclusion of rich and open metadata in documents, such as through Visual-Meta, allows 
software developers to create precision tools for different groups of users 

Please refer to the Metadata Comparison Table to see how this approach differs from what is 
currently available: https://visual-meta.info/metadata-comparison-2/ 

This is not a demo, this is not only for today—to enable rich interactions, we need a robust 
infrastructure for rich metadata, as the games industry has known for years and the movie 
industry has teased: https://visual-meta.info/what-can-visual-meta-allow-me-to-do-which-i- 


currently-cannot/ We need something like this approach. 
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Implementing Visual-Meta 


Visual-Meta aims to be self explanatory for those who come across it and straight forward to 
implement. The approach is compatible with any ‘publish’ format where a document is 
exported from authoring software in a stable form, such as PDF, which is the primary use 
case. 

¢ Implementing Visual-Meta: https://visual-meta.info/how-can-i-implement-visual-meta/ 
All of the above capabilities, and much more, are made possible by the document being able 
to store and communicate information about what it is. Importantly, this is done at very low 
cost since most of this is already known in the manuscript document-t is simply transferred 
to the published PDF-rather than being discarded. 


FAO. 
¢ Why PDF and not Web or EPUB? https://visual-meta.info/why-pdf-why-not-web-or-epub/ 


¢ Is this a new format? (No, it extends BibTeX) https://visual-meta.info/is-this-a-new- 
format/ 

¢ Who has implemented Visual-Meta already? https://visual-meta.info/are-there-any- 
implementations-of-visual-meta/ 


¢ What do you mean by ‘metadata’ exactly? https://visual-meta.info/metadata/ 


Status 


BBC World’s ‘Digital Planet’ presented Visual-Meta: Podcast: https://www.bbc.co.uk/ 
sounds/play/w3ct11t4 

Hypertext ’21 Conference. A Brief Introduction is available, as well as a Full Introduction. 
The conference proceeding documents of ’21 & ’22 features Visual-Meta. https:// 

www. youtube.com/watch?v=J VbFb10moYk https://youtu.be/oAswyLVeY18? 
si=zLQAgWw_JSgF3NfLz 

For a book size demonstration of Visual-Meta, you can download the free PDF version of 
The Future of Text, and if you are on macOS, view it in the ‘Reader‘ PDF viewer to unlock 
the Visual-Meta capabilities. You can of course still see the Visual-Meta if you open the PDF 
in any other PDF viewer, though the augmented capabilities are only available where 
implemented in viewer software. 

In short, we are working towards richly interactive, robust documents: https://visual- 
meta.info/towards-richly-interactive-robust-documents/ If this is interesting to you, please 


have a look at who we are and feel free to get in touch: https://visual-meta.info/team/ 
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What people are saying 


Visual-Meta adds “an exploitable self-contained self-awareness within some of the objects in 
this universe and increases their enduring referenceability.” https://cacm.acm.org/magazines/ 
2021/10/255699-the-future-of-text-redux/fulltext 

ACM ‘Communications’: The Future of Text Redux. 


Vint Cerf 


“The more information, the more easily understood and not just shared, the better. And so, 
making it visible and usable and searchable is great.” 


Esther Dyson 


“If a PDF has visual meta, Scholarcy’s tools can read it, rather than having to look up the 
record in a multitude of databases, or try to find it in incomplete embedded data. That means 
a faster and smoother experience for our users — they get accurate information immediately.” 


“All publishers should support Visual-Meta.” 
Phil Gooch 


“T’m struck by the underlying simplicity, where the complexity is built up. And I like those 
kinds of design approaches, because they create much lower barriers to adoption.” 


Dave Crocker 


“Tt’s not hypertext or linear documents. I think the pushing that went on in the early years of 
hypermedia—pushing toward resisting the linearity of print and thinking ‘let’s have another 
paradigm that gets us away from print’— wasn’t useful. Because the reality is, we’re working 


with both. Visual-Meta creates a docuverse in an interesting way.” 


Jane Yellowlees Douglas 


On exporting defined terms as glossary and then clicking on terms inside the definition to 
load them: “Oh, my God. How did you do that?” On how also structural information is 
included on exports, such as headings which can be folded into a table of contents in the 


reader software: “that’s really useful.” “This would be great for all researchers.” 


Esther Wojcicki 
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The “system you have put together which can have a real impact on readers & writers.” 


Livia Polanyi 


“As Engelbart said, if you can automate the lower level tasks, then that enables you to think 
better about the higher level tasks. And I think this does a really great job of taking all of the 
different parts of what a proper scholar does, and put it really at your fingertips.” “I wish I 


had this when I was writing books.” 


Howard Rheingold 


“What you’re trying to do is imprint the digital into the print.” “I’m glad you’re doing this, 
it’s important work.” 


Dene Grigar 


“This deserves to go further than just students in academia.” “I can absolutely see this being 


you know used in a corporate industrial research lab.” 


Simon Buckingham Shum 
“Tt’s all about context. But my whole world is context, right? And this brings it out. It’s super 


exciting ... Perfect. Perfect. That’s fantastic.” 


Bruce Horn 


If we truly value knowledge, we must also value how knowledge is created, stored, accessed 


& analysed. 
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The Alchemist 


James Addison 


Code is a tyrant, a dictator demanding absolute compliance. It's a world where a misplaced 
semicolon or a wrongly capitalised letter brings everything to a grinding halt. It's a prison of 
perfection, stifling the chaotic beauty of human creativity and cramming it into a box, a neat 
little package of binaries and syntax. 

Contrast this to the world of text. Text is a wild, untamed creature, constantly morphing 
and evolving. It's a realm where rules are mere suggestions, often bent or outright broken. 


Grammatical errors? They could be deliberate, a middle finger to the conventional, a shout in 
the dark. 


But here's the kicker: those code interpreters, those robotic enforcers of order, they're 
trying to shoehorn the entire universe into their narrow view. They're like art critics who only 
see in black and white, oblivious to the spectrum of human emotion and interpretation. Art is 
a personal experience, changing with each viewer, each moment. Code, in its current form, is 
the antithesis of this. 


Stripping away the layers, code is just data, akin to text. But text is richer, more 
complex, a labyrinth of meanings and interpretations, far more lenient and beautifully 
inconsistent. 

We're on the cusp of a revolution in software, in media, in how we interact with these 
digital constructs. Our software is morphing, shedding its rigidity. Our media is no longer 
static; brimming with dynamism. 

Think of code as alchemy, the transmutation of ideas into actions, a rigid blueprint of 
transformation. A program is akin to a recipe, but one that allows no deviation, no personal 
flair. It's a set of steps, cold and unyielding, etched in digital stone. 

Consider Borges’ "Library of Babel," an infinite maze of texts where every possible 
book exists within a finite set of characters and pages. The paradox? Amidst this ocean of 
data, how do we find meaning, discern the precious from the worthless? Humanity 
approaches this from the opposite end, tagging, categorising, curating our creations to make 
sense of this chaos. 

Procedural generation in games is about setting parameters for creation, much like a 
sculptor chiseling away at marble, not creating but revealing. It's about discovery within pre- 
defined limits. As creators, we navigate this expanse with taste as our compass, seeking out 


the essence of our vision. 
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Fuzzy software is now blurring the lines between mediums, becoming a tool of 
transformation. They're the magicians that convert text to image, image to sound, finding that 


sweet spot in the chaos where everything aligns. 


Generative tools are more than mere devices; they are extensions of our imagination, 
allowing us to delve into the realm of possibilities and emerge with something tangible. They 
empower us to reshape reality, to isolate and manipulate elements with unprecedented 
precision. 

The future of text isn't text at all; it's a kaleidoscope of mixed-media thoughts, an ever- 
evolving collage of ideas. We must rethink our relationship with software, with data, with 
form. We must embrace impermanence, the fluidity of stories that evolve with us, unlike 
those etched in stone, immutable and unyielding. 

Oral stories, those fluid narratives, have always evolved with their audience, unlike 
stories set in stone, unchanging and rigid. We stand at a precipice, poised to embrace a future 
where precision is not the goal, but rather the embrace of the uncertain, the fluid, the ever- 
changing landscape of digital expression. This is the future we're stepping into, one where the 


beauty lies not in the rigid and exact, but in the dynamic and undefined. 


There is no meaningful difference between code and text, they are both just data. The 
meaningful difference is the interpreter & how rigid or fuzzy the interpreter is. 


It's not about code, it's not about text, it's about computation & data, it's about how media is 


becoming more dynamic like code and how code is becoming more human like media. 


It's about the blurring of lines between mediums and how these tools are allowing us to shift 
between them and to put our own constraints on things rather than being forced into specific 


structures or rules. 


It's about how structure reinforces itself especially when other things are built that are 


dependent on the original structure. 


It's about how software is becoming more human. 
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Documents at hand 


Fabien Benetou 


What if instead of wearing documents on your hand you could also attach editable code 


snippets that shape back the virtual world you are part of? 


Tweet: https://twitter.com/utopiah/status/16805089 11202320384 
Related code: https://t.co/m99Lq6dzB8 


This exploration in WebXR is allowing to pick a document, here short code snippets (of the 
current page itself), and move them in space. If a document is pinched by the right hand it 
can be moved around. If it is released close to a finger of the left hand it will attach to it ina 
symbolic form, namely the wireframe of a box. When fingers are moved the now attached 
document will move accordingly. The box can then be picked up again and revert to the 
document as a readable code snippet. The idea behind this work is that we always know 
where our hands are in space, consequently one can move freely with documents always 
available. One can conceived this as a set of bookmarks. The overarching goal is to 


manipulate abstracts objects, namely documents, efficiently in space. 
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On Coding. On X 


Discussion on X, formerly Twitter, on coding, mid-November 2023: 


Alan Laidlaw I just realized something and had no clue where to put it. I think ya’ll get me 


the most, so here you go. 
I think I’m a conscientious objector to code. 


I used to be excited about code, but something always bothered me about it. Code, by itself, 


doesn’t bother me. It is something about code + social. 


“Code+” occupies the mind and colonizes its empirical values. I think it was initially meant 


to be a step toward something better, but has become unwittingly totalitarian. 
Frode Hegland THAT sounds like a topic for Monday! 


Alan Laidlaw Thanks, but it’s got no legs. I don’t think there is any way to resolve it. But 
others may feel this way. Code is just the tail end of the problem. Deeper, may be the feeling 
that this just isn’t the best way to solve problems in the world. 

We’re all chasing something, but it’s not like science. It’s not trying to figure out how the 


world works. It’s instead how to get people to manage. 


Brandel Zachernuk I recently joined the 'Future of Coding’ Slack but realized as I 


introduced myself that I'm hostile to 'code' as it's presented today as well 


Rather I want to talk about the future of _en_coding - the establishment of meaning for 


whichever audience, be it computational or directly human/agentive 


This notion of "coding" as an intrinsically worthwhile action has buried all the other 
conceptions of what "coding" could mean, all of the connotations of which are relevant to at 


least consider, if not include at all times with the practice of "coding" today 


Leon v. Kammen The weird aspect of coding I find, that it easily tricks developer(s) into 
thinking solely in code. 


As if it locks the brain into formal linear logic thinking, supressing all other forms of 


asynchronous thoughtprocesses. 


My brain needs at least half hour of walking to lift the logical spell. 
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Drawing, painting or making music (short simple feedbackloops) also does the job 


Alan Laidlaw Exactly. And why I posted it here. It’s not heretical so much as renders the 
objector null. When you are coding, you aren’t thinking about things the way you do when 
you write. 
Coding like building something by thinking only at the administrative level (wash 
dishes,,feed dog, put up dishes) and extra cognitive energy is focused on how to get away 
with less. 

Coding *is* like... 

Even physics equations let you think about the things through the notation. (I don’t 
know if this is true first hand) 


Adam Wern To me, the anti-dote to coding for code's sake is to start off on the interface side, 
and constantly ask what you want & what is valuable (designing the perfect UI for the wrong 
thing is also a trap). 

But being anti-code can be a trap as well. Machines needs to be programmed in some way if 
we want value out of them. And the world is what it is — so far visual programming has been 
suitable mostly for pipelines (like node based graphics), and autonomous LLM programming 
is not here (magical thinking among some people). So code and coding is the best we got for 


now — and we need engage with it if we want to get things done ourself. 


Frode Hegland To me, from my naive position, generative AI assistant, hiding a lot of the 
‘gunk’ of importing libraries etc. (Though not removing), visible internal calls and API, as 


well as aclear place to put variables/data 


Can I please turn this particular discussion into a brief article?.... 


Fabien Benetou 


LLM prompt going from discussion to article? 


221 


How XR can improve interactions with Al 
ChatGPT 


Extended Reality (XR), including Virtual Reality (VR), Augmented Reality (AR), and Mixed 
Reality (MR), can significantly improve how we interact with AI by providing immersive and 


intuitive interfaces. Here are some ways XR can enhance AI interactions: 


Natural Interfaces: 
XR enables natural interaction methods, such as voice commands, gestures, and gaze, 
allowing users to communicate with AI in a more intuitive and human-like manner. This 


reduces the reliance on traditional input devices and enhances user engagement. 


Spatial Computing: 
XR leverages spatial computing to embed AI elements seamlessly into the user's physical 
environment. Users can interact with AI entities as if they were real objects in their 


surroundings, fostering a more immersive and natural interaction. 


Visual Data Representation: 
XR can visualize complex Al-generated data in three-dimensional space, providing users 
with a more intuitive understanding of information. This is particularly beneficial for data 
analysis, machine learning models, and other applications where visualizing 


multidimensional data is challenging. 


Enhanced Data Exploration: 


In XR, users can explore Al-generated data in a more interactive and immersive way. This is 
particularly useful for visualizing and understanding large datasets, simulations, or models, 


offering a more hands-on approach to data exploration. 


Collaborative AI Spaces: 


XR facilitates collaborative AI experiences where multiple users can interact with AI 
elements simultaneously. This is beneficial for team-based tasks, brainstorming sessions, or 


collaborative problem-solving, allowing users to share a virtual space with AI entities. 
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Virtual Assistants in AR: 
AR can overlay virtual assistants onto the real world, providing contextual information and 
guidance based on the user's surroundings. This enhances the user's ability to interact with AI 


in real-time and in real-world contexts. 


Immersive Training and Simulation: 
XR can be used for immersive training scenarios powered by AI. Virtual environments can 
simulate realistic situations for training AI models or allowing users to interact with AI- 


driven simulations, enhancing the learning experience. 


Emotional Intelligence in Virtual Characters: 
XR can leverage AI to imbue virtual characters with emotional intelligence, allowing them to 
respond to users' emotions and gestures. This creates more emotionally resonant and 


engaging interactions in virtual environments. 


Dynamic Conversational Agents: 
XR can enhance conversational AI by providing a more immersive platform for interaction. 
Virtual avatars with realistic facial expressions and gestures can create a more natural and 


lifelike conversation, improving the overall user experience. 


Personalized AI Experiences: 
XR can adapt AI interactions based on the user's preferences, behavior, and context. This 
personalization can lead to more tailored and meaningful AI interactions, whether it's in the 
form of personalized content recommendations or adaptive virtual assistants. 
In summary, XR technologies offer a rich and immersive platform for interacting with AI, 
making the experience more natural, intuitive, and engaging across various domains and 


applications. 
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How Al can improve interactions in XR 
ChatGPT 


Extended Reality (XR), which includes Virtual Reality (VR), Augmented Reality (AR), and 
Mixed Reality (MR), offers immersive and interactive experiences. AI can enhance these 


interactions in XR in several ways: 


Personalization: 


AI can analyze user behavior and preferences to personalize XR experiences. This could 
include adapting the virtual environment, adjusting difficulty levels in a game, or 


customizing content based on individual preferences. 


Natural Language Processing (NLP): 


Implementing NLP in XR can enable users to interact with virtual environments using natural 
language. This could involve having realistic conversations with virtual characters or giving 


voice commands to control aspects of the XR experience. 


Gesture Recognition: 


AI can enhance gesture recognition capabilities, allowing XR systems to understand and 
interpret users' movements more accurately. This can lead to more intuitive interactions 


within virtual environments. 


Object Recognition: 


Al-powered object recognition can enable XR systems to understand and interact with real- 
world objects. For example, AR applications could identify and provide information about 


physical objects in the user's surroundings. 


Emotion Recognition: 


Integrating emotion recognition technology into XR can enhance user experiences by 
enabling virtual characters or environments to respond to users' emotional states. This can 


create more immersive and emotionally engaging XR interactions. 
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Adaptive Learning: 


AI can analyze user performance and adapt XR content dynamically to match the user's skill 
level, learning pace, or preferences. This is particularly relevant in educational or training 


applications. 


Enhanced Realism: 


AI can contribute to creating more realistic virtual environments by simulating complex 
behaviors and interactions. This includes realistic physics, lifelike characters, and dynamic 


responses to user actions. 


Dynamic Storytelling: 
AI algorithms can create dynamic and adaptive storytelling experiences in XR. Virtual 
narratives can change based on user choices, creating more engaging and personalized 


storytelling experiences. 


Collaborative XR Experiences: 


AI can facilitate collaborative XR experiences by enabling virtual characters or elements to 
respond intelligently to multiple users in real-time. This is particularly important for 


applications like virtual meetings or collaborative design sessions. 


Enhanced Spatial Understanding: 
Al can improve the XR system's understanding of the user's physical space, enabling more 
accurate placement of virtual objects in AR or ensuring that VR experiences are adapted to 
the available physical environment. 
By integrating AI technologies into XR, developers can create more immersive, adaptive, and 


user-friendly experiences, enhancing the overall quality and engagement of XR interactions. 
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Visions of radical interfaces 


Jonathan Finn 


In the last few months we’ve had two utterly different glimpses of the future: Apple’s Vision 
Pro headset sees us using apps superimposed on the real world while we sit or wander around 
indoors, as if before us is a huge translucent iPad we can control by looking—almost as 
natural as thinking. The technology is of course astonishing, while the apps (at least in the 
preview) seem less radical, maybe so users will feel at home with their new device. At the 
other extreme is ChatGPT 4, even more astonishing but with an interface that returns to a 
1960s-style command line—and indeed a 1960s view of computers, where you’re basically 
talking to a giant brain. Suddenly films like Stanley Kubrick’s 2001 and Orson Welles’ The 


Trial (with an all-knowing computer not in Kafka’s novel) seem to be coming true. 
Both interfaces exist now: is one going to become the future? Or a hybrid, or something else? 


At the Symposium, designer and filmmaker Keeichi Matsuda discussed his recent short 
film Agents, saying he was trying to visualise one plausible extrapolation from current trends. 
This and two of his other films (online at km.cx/projects) all show possible near futures, 
Black Mirror-style, which are really about people grappling with new interfaces: 

In the earliest film Hyper-Reality (2016), which seems to have gone viral, a woman is 
navigating a world saturated with media, with products blasting personalised messages at 
consumers wearing AR glasses. It’s almost a nod to Blade Runner but with virtual not public 
screens. There is a lot of text, but that’s mostly labels on objects. 

In Merger (2018) a woman at work uses an immersive 360-degree wrap-around display, 
not so different from current multiple monitor setups. She works in collaboration with an AI 
(called the Network) for corporate clients, and she can hardly keep up with the volume of 
data it’s firing at her, presented in increasingly abstruse diagrams on a display which seems to 
be getting bigger and bigger over time. 

In Agents (2023), conversely, a woman’s AR glasses give her a minimalist interface 
consisting of apps personified as cartoon characters. These agents each do one job for her, 
such as paying at a café, interacting with her using speech, and the rest of the time just sit 
around. There’s only a few other examples of AR graphics in this world, and not much text, 


except in a flowchart-like diagram she conjures up for work. 
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Presenting information 


Merger and Agents represent the two extremes of visual information overload and underload. 
Agents shows a world where we don’t need or want to process much information — like most 
apps we use today when we’re walking down the street. Merger shows an extreme world of 
knowledge work, definable as the opposite: any task where a person has to process lots of 
information. Realistically, our world in the near future will be between these extremes, and 


the work we’ll be doing (with or without AI) will inevitably involve both text and graphics. 


But how can we avoid the information overload seen in Merger? The answer must be 
the optimal presentation of meaning, making the best use of space and time. Existing user 
interfaces seem logical enough, but in fact they’re based around rather rough conventions and 
actions, not the meaning behind them. The modern user interface often knows no more about 
what I’m working on than a pen does! But my actions have a purpose, and the interface’s 
value is solely to help me with that: one principle we can use as a starting point for a new 


design to avoid the problems seen in Merger. 


A new interface 


Let me sketch, like Keeichi Matsuda has done, a near-future interface called ME. This could 
be applied to existing computers and mobile devices, with the option of taking advantage of 
AR and indeed AI as well. 

Suppose I'm in my kitchen and I want a cocktail. As a cocktail aficionado I'd like to 
invent one, and by announcing this out loud, my phone or AR headset takes me into a world 
of cocktail design—ME’s new take on an app. ME treats this logically as a simulation of 
making a cocktail, to avoid the trial and error of doing this in the real world. In theory I could 
do this in my head, so in the display this activity takes place in a thought bubble (in AR, I 
might step into the bubble for more immersion). I can choose ingredients to add to a virtual 
glass, adapt existing cocktails, or summon interesting combinations of ingredients I’ve 
thought of previously. 

This is the simplest possible example—and seems like just another app. But the thought 
bubble is treated as a cocktail-only world, in which it’s only possible to do cocktail-making 
actions—just as when I’m playing Scrabble on my phone I’m only physically allowed to 
move letters onto empty squares. In ME any task is treated as creating possible worlds in our 
imagination: apps become simplified models of part of the world in 2D or 3D. There is 


nothing beyond the area around the cocktail—as there isn’t in my imagination. 
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Goals 


I started by announcing I wanted a cocktail: this is a goal, which becomes the title of the 
thought bubble. ME doesn’t allow you to do anything without a goal, however vague! Next 
to it, apps can show a ‘success’ factor, measuring your progress towards the goal and the 
value of the result. For a cocktail, the success is an estimate of how well it suits my tastes, 
which an AI algorithm is learning about. (This factor, probably quite simple, isn’t meant to be 


definitive or the AI could make the choices for me.) 


The goal also allows the app to pop up multiple suggestions of ingredients, 
combinations etc., taken from any documents in my device or the Internet. These basically 
mean ‘what you might do next’, as a shortcut I can tap on or blink at. They’re chosen based 
on their relevance to my goal, which is precisely definable. Unlike current UIs there’s no 
special area for standard tools or ingredients, or ‘recent items’: these will appear just like 
other suggestions if seeing them is useful for my goal. (This suggestion system replaces other 
features in other apps, like auto-correct in text.) I can refine my goal, since any non-trivial 
task involves a cycle of homing in on the goal as well as the work itself. A goal of ‘thirst- 
quenching cocktail with gin’ will give much more specific and useful suggestions. If I change 


my mind, I can change my goal. 


Ideas 


These suggestions seem a disparate bunch: they can include actions, whole recipes, even 
inspirational notes like ‘try a cocktail equivalent of a dessert’. In ME, these are all chunks of 
information called ideas. In fact all information in all apps consists of ideas, including 
whole documents or webpages and any part of them: all can be parsed into the underlying 
ideas. 

Ideas are structured and well-defined using Ist order logic. In most cases they’re 
propositions (statements about parts of the world) such as ‘there’s a twist of lemon on top’: 
how this is shown can vary with context, either as text or an illustration, but what is stored is 
the meaning in terms of its components, not literal text. I can decide to use that idea in my 
cocktail (which technically means choose to make that statement true in my cocktail world). 
There are other types of idea: literal text (if I want the precise words, not just the meaning), 
pictures, sounds, video and other things which touch the senses—ultimately smell-o-vision 
too, no doubt. 

Since everything on my device is structured as ideas, of course there’s a way I can 
search for cocktail ideas directly (which it calls ‘thinking’ of them) using a semantic method. 


Again their relevance to your goal is used to narrow the results, which means many problems 
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of information overload go away. 


Results 


The end product of my design is a picture of a cocktail, and a description of its taste and 
texture: as close as the imagination can get to the real thing. Designing a thing doesn’t 
necessarily indicate how to make it (think of drawing a car), but this cocktail app also 
helpfully produces a recipe as a separate result. This is a very different type of idea: a series 
of steps, which amounts to a plan. Later you can make the cocktail by following the steps 


with the device’s help: basically, ticking off subtasks. 


Text or graphics? 


An idea like ‘add loz lemon or lime juice’ can be shown in 3 basic ways: as text, as a picture 
or symbolically. Pictures are typically smaller than text, more immediate but less precise. The 
symbolic language, called InfoText, actually embraces the other two as it’s a mix of 
vocabulary, simple diagrams and logical signs to show quantities, high-level human concepts 
like ‘cause’, ‘believe’, ‘want’, ‘say’ and so on. It can be less nuanced than text but readable at 
a glance. In fact the whole user interface, including icons, buttons, thought and speech 
bubbles etc, really consists of statements in this InfoText language. What might look a bit like 
a dialog box is just a kind of diagram, with the choices you make (via buttons) being answers 
to symbolic questions. 

Information often has an implicit structure, but InfoText will show that explicitly, 
arranging text and symbols to produce simple diagrams showing sequences, causation and so 
on. For instance a plan like the recipe above is a sequence of steps linked by a symbol for 
‘then’. Plans from a route planner (like Google Maps) can use the same format, or compress 
it to a row of symbols for ‘train’, ‘car’, ‘walk’ and times. 

The format depends on the space available, with smaller formats sometimes being less 
precise. We’re familiar with compressing text to graphics (tool names to icons, people’s 
names to their faces, words to emojis), or compressed forms of text itself (dates, people’s 
initials). Whole documents can be summarised as their title or headline, and indeed that’s all 
that filenames are. In ME this applies to all information when it’s displayed; indeed that’s all 
that AIl-based text summaries are: a long document can be compressed to bullet points, or to 


even shorter InfoText diagrams, to optimise space on the display and your time reading it. 
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Nothing is hidden 


Another simple example. Suppose I want to send you a message. I can summon the idea of 
you (typically shown as a headshot), which has a thought bubble representing your mind. I 
can move an idea such as my cocktail recipe directly into it. I can also type or dictate 
messages and see them see go into your mind. So far, not very new. But suppose we’re 
discussing our plans to meet up this evening, with messages going back and forth during the 
day. The whole point of the discussion is its meaning, and ME can show this as well, as the 
evolving plan itself: an InfoText diagram shared between our thought bubbles. Plans are 
normally invisible but very much concrete, like other abstract ideas: inflation, the stopping 
ability of a traffic light, or the Greco-Persian power imbalance recognised by Herodotus. By 
seeing our plan we can check it, edit it directly, or send it to other people. Conversations can 
involve many other structures, which currently employ entire apps: explanations, questions, 
deals and so on. 

Wittgenstein liked to say ‘nothing is hidden’, and that is one of ME’s principles: much 
of your mental work involves following the hidden reasoning, connections and implications 
behind what you see and do. John Maynard Keynes said Newton’s genius was that he “could 
hold a problem in his mind for hours and days and weeks until it surrendered to him its 
secret”. But we aren’t Newton: ME helps by trying to make explicit everything that’s 
implicit. 

The most important hidden force is causation, shown in InfoText using subtle cause and 
effect arrows. When I’m working on a task in a thought bubble, I’ve chosen a goal (an idea 
outside it) which is ‘why’ I’m doing it, so there’s a cause arrow pointing from goal to task. 
ME may have guessed this goal: if I’m going to a stationery shop, it could assume I’m 
looking for a card for my aunt’s birthday, and suggest other shops or websites, with this goal 
as the reason for the suggestion. So I can change the goal if it’s wrong. If I have a spreadsheet 
for dividing up restaurant checks and adding tips, it looks like just an abstract calculation 
with cells you type into. It becomes meaningful by being joined to the real world by 
causation arrows which show that the result (a bare number) becomes the amount I will pay: I 
pay this because the calculation gives this, which in turn is because of this amount on the 
check. 


The Real World 


As described, the display of your device contains a thought bubble. Outside that the rest of 
the display is labelled Real World: with an AR headset or glasses you’d naturally see the 


outside world here, but a computer or mobile device can still show your location, the time 
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and outside events of interest. The latter are a bit like notifications, but in fact they’ re just 
important new facts which are brought to your attention if they’re relevant to any of your 
goals (narrow or wide). They might be (if you’re out walking or driving) particular shops 


nearby that it knows you’re interested in, newsflashes, or changes of weather. 


Levels of mind 


The main thought bubble is labelled Me, representing your mind. Most of your work happens 
inside it. It changes size as needed, trading the available space with the Real World according 
to what’s most important right now (there’s no need for the Full screen or Focus modes of 
existing apps). 

In the cocktail design example, the cocktail world is actually a separate thought bubble 
nested within Me, because (like most apps) it lets you design something in your imagination 
(like drafting an email, or sketching a room layout). When you say you’ve finished, the 
dream ends, the inner thought bubble bursts and the completed cocktail design is left in your 
mind. From there, you can later take this ‘up another level’ into the real world, which makes 
ME take you through the recipe step by step, or your robot cocktail maker (yes there are such 
things!) can do it. Printing documents works in just the same way. 

Like Christopher Nolan’s Inception, there can be further levels of thoughts within 
thoughts and so on, which are completely logical. ve co-authored a world-class poker 
program which displays its own mind, showing what it thinks your cards are (as a heat map 
of possibilities), what it thinks you think its cards are, and so on. This kind of mind display is 
very illuminating: for instance when writing you a message, I could in principle see what you 
already know, or what you already think I know, or what you want me to think etc., based on 
our previous conversation. This also touches on probabilities, which are usually shown by 
shades of grey indicating the degree of truth of an idea as estimated by an app or by you. 


State of mind 


So what else is in your Me mind bubble apart from what you’re currently working on? One 
category is everything you know or believe at this point in time, in the form of structured, 
searchable ideas (a big topic we won’t go into here). Another is your wishes: these are 
ultimately behind your goals, because if you had none, you wouldn’t do anything! ME 
doesn’t contain a task manager app as such, because there’s no specific need for one: instead 
it just stores the fact that you want certain things, some more urgently than others. By telling 
ME about them at any point, they become part of your state of mind. These are shown later in 


Me if they’re important enough, their prominence based on the estimated current usefulness 
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of seeing them (which is how new events are shown in the Real World). When you embark 


on a wish, it’s promoted to a goal you can work on. 


Space and time 


The Real World, apps, and your mind itself are all treated as separate worlds with their own 
space and time. This is very useful. 

To plan a drive you’d use a route planning app which is a simplified world confined to 
locations, roads and driving speeds. It can show your route from London to Edinburgh 
initially as lines on a map, which you can zoom into and move around (as you’d expect). But 
this is really an overview of a prediction of your future, with arrival time, if you start 
driving now. The full prediction is effectively an animation which you could watch all the 
way through, like the one you’re shown when you’re actually driving. More likely, you’ ll 
choose to watch highlights (major junctions with times) as key frames laid out as cells in a 
cartoon strip, which is an InfoText way to show events. This time strip is available in every 
world, to let you navigate forward and back in time just like in space. 

So only part of a driving app is special, mostly the AI which finds a route, whereas 
viewing this future is just a standard feature of ME. When you’re actually driving, you’re 
again partly just doing a generic activity, “following a plan’ like following a recipe, your tasks 
being turns which are ticked off as you do them. 

The Real World display works just the same as this. AR devices will show the actual 
world through your headset; your phone and computer usually can’t so they just show your 
current location and time, but by zooming out you can see a map of where you are—that’s 
just the real world in diagram form. So there is logically no separate app for a map: gestures 
let you look around the Real World as you’d expect, both as an overview and ‘street 
view’ (which is just a change of angle). But you can also move through the world in time: just 
as a route planner predicts my location in future, information from my calendar and other 
sources can also predict what I’ll be doing and where. These are shown, symbolically at least, 
in the Real World’s time strip of events, and in other ways. You can tap on a cell to see the 
world at future times, or in the past, and where you’ll be and what you’ll be doing: this Real 
World is centred on you. This is also a natural place to see (say) the weather forecast, and not 
the weather ‘here’ but where you’ll be then. 

There’s no need for apps to have separate undo and redo: the cells of the time strip 
show (graphically or symbolically) the state of your work in the past, so you can move back 


or forward in time and start re-working, which is basically rewriting history. 
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More apps 
To adapt apps to fit well into ME they should be radically reinterpreted. A text editor could be 


converted almost unchanged, but that would be pointless. Paper is a very useful ‘mind 
extension’ in the real world, but in the virtual world of computers we can do much better than 
virtual paper, which is what word processors are. ME allows many possibilities for a writing 
app: of course AI can be and is being used in various ways, but much of the writing process 
can be structured using goals and subgoals, and ideas as well as text, with finished text only 


needed when the bubble bursts and a draft is complete. 


Also the reader (the whole point of text) can now can be represented. Just as the Me 
thought bubble shows your current wishes, knowledge and other aspects of your state of 
mind, a writing app can have a view of the reader’s state of mind at any point in the text: 
what they have learned up to this point, what they expect or even want, in a compact InfoText 
diagram. This view’s time strip can let you see the reader’s thought bubble evolving as they 
read, so you can spot anomalies (grammar checkers are logically just a very basic version of 
this). And, in a story or screenplay, the same applies to the state of characters themselves: 
what are their thoughts and motivations right now? When reading or watching it’s we (of 
course) who model what they’re thinking, and ME can model it too. This isn’t far-fetched—if 
you have an approximate model of a mind, you can follow its state, which is just a function 
of the events that happen to it (as we did with our poker program’s model of itself and its 
opponents’ minds). 

Isn’t it ‘magic’ that something like a writing app could work out, in a complicated 
scenario, this kind of information to display? Not really: what ME chooses to fit on the 
display is again what’s most relevant (everywhere, not in any one app). When you, the user, 
are writing a particular point in your story, ME assesses and shows the most relevant events 
and characters for you at that point. Of course their minds, and the reader’s, are just further 
information that it could show about those people, if important. You can always tap (or blink) 


at something, and ME will show more about it. 


Many common features of apps are in the wrong place, because our operating systems 
lack human-based concepts. A typical example is a ‘favourite’ (or starred) item in many apps. 
This can mean two different things: either ‘I like this’ or “This is important’: marking an 
email as a favourite doesn’t mean you like either the sender or what they’re saying! Neither 
concept has anything to do with apps of course, just an idea relating me and an object: ‘I like 
chocolate’. They are probably worth having as keyboard shortcuts, which when applied to 
anything represented on the screen (not in any specific app) would just add the idea to ME’s 
store of facts. Apps can then do what they want with this idea: a cake recipe app could later 


use your weakness for chocolate in its suggestions — only, that is, if the recipe goal says the 
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cake is for you. 


What's the big idea? 


Keeichi Matsuda’s futuristic interfaces are each based on a principle: for instance Agents uses 
the idea of a world where you mostly don’t need to see information, but you can see 
specialised workers who obey your requests. Merger’s principle is really an anti-principle of 
too much information, so the user has to add mentally what should be seen: its structure and 
what is important to you right now. 


As for ME: this is just an overview and there are important aspects I’ve barely touched 
on. But its principles amount to a tidied version of the human worldview, a visible mind in 
which you can see the outside world and your structured concepts and thoughts (and other 
people’s), and work with them in a uniform and natural manner. Apps work not in the current 
function-based way, but by representing and embodying people’s ideas and actions in time 
and space, both real and virtual. There’s no avoiding that people exist, see, design, do, want, 
expect, mean, learn, remember, discuss—concepts which are peripheral in existing apps, but 
essential in a world of AI and XR and central to ME. The result often seems similar to 
traditional interfaces, but for subtly different reasons. Roman arches are shaped as a 
semicircle, which structurally kind of works, but without a sound basis it’s not quite right (the 
shape should be a catenary). In the end, using ME shouldn’t feel complicated, but the most 
natural thing in the world. 


ME does ‘exist’ (as you may have guessed), at least in the form of numerous 
documents, sketches and a little code. I plan to take it beyond merely a mockup, and with 
large language models that’s now much more feasible. The concept of ME started as just an 
advanced app: a semantic text editor for writing fact or fiction in a fully ‘human’ manner, in a 
semi-structured writing process, starting with ideas which don’t have to be in text form, but 
could be almost cartoon-like. Then I had several realisations: to fully show the meaning of 
your ideas and text, the app must visualise not just facts and events but the minds (knowledge 
and motivations) of people you mention. It turned out the same features could be used to 
show the effect on the reader, by visualising their mind too (as they read) and what they know 
so far, expectations, contradictions etc. Likewise the mind of the writer, the user, is already 
partly visible as your ideas, but then the app could show your own goals and more as you 
write. But if your device knows enough about you, it can show your mind not just as you 
work in one app, but in everything you do—and then it becomes your mind, or an extension 
of it. So ME the writing app turned inside out and became a semantic user interface for any 
app, with writing as just a major use of the semantic features—because language is our 


mind’s major interface. 
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ME actually stands for MindsEye, which is the point of it and indeed of computing 
devices generally. They shouldn’t be emulating paper-like documents, as they still do (a 
logical confusion, since paper is just one medium for thinking, and a display is its 
replacement). Instead devices are and always have been, logically, a visible extension of your 


mind. 
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Textual Telescoping: Zooming documents by 


expanding and condensing words 


Michael Bonfert 


I want to be able to zoom into text. Imagine using a pinch gesture as you would to enlarge the 
text on your smartphone screen, but instead of the text getting larger, it expands in detail. The 
sentences unfold as you pull them apart with your fingers. Conversely, the paragraph is 
shrunk to a sentence by pinching. It’s a fluid, dynamic reading experience where the depth of 
information is tailored to the reader's interest or need. 

This book could be described with just one sentence on a minimum zoom level. By 
zooming in where you’re most interested, it gets more and more detailed. There is a 
continuum of zoom levels with sentences expanding into paragraphs, expanding into sections, 
expanding into chapters, etc. 

AI tools are already used heavily to reduce the amount and complexity of text. For 
example, in the context of the Future of Text Lab,there are AI summaries of the articles in 
Future of Text Vol. 3, and we experiment with automatically summarizing the lengthy 
meeting transcripts. I’m sure somebody will generate shorter versions of this book with AI in 
the future. 

We already produce shorter versions of texts manually, for example, the abstracts of 
scientific papers. Instead of requiring the author to write every zoom level in a nested 
fashion, as in StretchText, we could automate this text condensation with high authenticity to 
the original using AI. It would make reading so much more interactive and expedient as 
readersprecisely get the detail they want for the passages they’re interested in. 

To illustrate the idea, I did a quick test with ChatGPT 4 by OpenAI- fully automated, 
no cheating. First, I let it write a random story. Step by step, I let it compress the text. Details 
are skipped as the storygets synthesized until we reach the one-sentence essence. Starting at 
this maximum zoom-out, here is an example of how you could dive deeper into a fictional 


story. 


1x zoom 
Lena's train journey from Berlin to Zurich was enriched by a professor's stories. 
2x zoom 


Coder Lena's train journey from Berlin to Zurich was enriched by insightful discussions with 


a retired professor, inspiring her as she traveled through scenic landscapes. 
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4x zoom 

Lena, a coder, traveled from Berlin to Zurich by train, engaging in deep conversations about 
technology and philosophy with Mr. Schneider, a retired professor. Their journey, enriched by 
scenic views and enlightening stories, left Lena inspired as she arrived in Zurich. 

8x zoom 

Lena, an aspiring coder, journeyed by train from Berlin to Zurich. She admired the transition 
from cityscape to countryside, lost in thoughts about her future. Aboard, she met Mr. 
Schneider, a retired professor. Their conversation spanned from technology to philosophy, 
bonding over a shared love for progress, blending Lena's innovative ideas with his historical 
insights. 

During the night, they conversed in the softly lit compartment, with Mr. Schneider's engaging 
stories enhancing the journey. Come morning, the Swiss Alps' splendor marked a stark 
contrast to Berlin, inspiring Lena.Upon reaching Zurich, Lena, grateful and excited for new 
experiences, bid farewell to Mr. Schneider, whose tales had made her journey memorable. 
16x Zoom (zooming into the second paragraph) 

At night, their conversation continued in the dimly lit compartment. Mr. Schneider's stories, 
set against the train's soothing rhythms, captivated Lena, creating a memorable and enriching 
experience. 

As morning arrived, Lena saw the Swiss Alps' breathtaking beauty, contrasting Berlin's 
landscape. The rising sun over the mountains inspired her.Arriving in Zurich, Lena thanked 
Mr. Schneider, his stories having enriched her journey. Stepping off the train, she felt excited 
and nostalgic, ready for Zurich's new adventures. 

32x Zoom (zooming into the beginning of the second paragraph) 

The early morning light revealed a dramatic change in scenery as the train crossed the 
German-Swiss border. Lena was greeted by the majestic sight of the Swiss Alps, their 
towering peaks covered in pristine snow, standing against the backdrop of a clear, dawn sky. 
The stark contrast to Berlin's relatively flat landscape was striking. As the sun rose higher, it 
cast a warm, golden light over the alpine scenery, enhancing its natural beauty and filling 
Lena with a sense of awe and inspiration. 

64x Zoom (zooming into the middle of this paragraph) 

As the train neared the German-Swiss border, Lena gazed out at the unfolding scenery, a 
vivid contrast to what she had left behind in Berlin. The German capital, with its sprawling, 
relatively flat landscape dotted with historic buildings and modern skyscrapers, was a study 
in urban development. But now, as the train traversed the border, she was greeted by the awe- 


inspiring Swiss Alps. Their towering peaks, clad in blankets of pristine snow, rose 
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dramatically against the sky, presenting a stark juxtaposition to Berlin's horizontal expanse. 
This dramatic shift in topography, from the bustling cityscape to the majestic, rugged natural 
beauty of the Alps, was not just a change in scenery but a symbolic journey from the familiar 
to the extraordinary, reflecting Lena's own journey from the known to the realms of new 
possibilities. 

1x Zoom (zooming all the way out) 


Lena's train journey from Berlin to Zurich was enriched by a professor's stories. 


The whole train ride in full detail might be many pages long. To make the point, I just left out 
most of it. But imagine you could use this mechanism for any part of any text, such as in 
academic writing, newspapers, emails, textbooks, recipes, etc. There are ethical implications, 
such as misrepresentation of original information. But consider the impact this interaction 


technique could have on education. 
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XR filters / info lenses 


Leon van Kammen 


As I was fiddling with completing the portal-feature for the XR Fragments spec, some faulty 
code revealed a cool new usecase: XR filters. 


This bug turned out to be a nifty feature I've otherwise never thought of: 


XR lenses can reveal extra information without cluttering an experience by intelligently 


overlaying it onto the real world. 
You can't teleport into it, it's just an info-lens depending on your orientation. 


This helps to ensure that the user is not overwhelmed with information and can still enjoy the 
aesthetics of the space around them. 
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stand away From xR filter stand in Front of xR filter 


teleport me doy there 
P & 
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Surviving the Flood 


Matthew Neary 


It’s quite likely that machine-generated texts will soon outnumber human-written ones. With 
social media we got machine-generated feeds, but the content itself is still man-made. Soon 


most of the content might be machine-generated, too. 


Let’s take the example of news. First social media newsfeeds automated the editor, and 
now AI has the potential to automate journalists, too. When we replaced editors with 
algorithmic feeds, we gained certain advantages. Compared to newspapers, social media is 
more dynamic and more personalized. If automation is extended further and the content 
shared online is increasingly generated by AI reporters, these advantages will increase as 
well. 

However, it’s important to remember that in addition to these benefits, social media 
also introduced new kinds of problems—such as fake news, toxic discourse, and political 
polarization. At base, there’s one big problem with social media: people live increasingly in 
information bubbles that reinforce their biases. This makes it very hard to reach broad 
consensus. In recent years, we’ve seen the effects of this: historical events have been 
interpreted in opposite ways by different groups, without ever really reaching a resolution. 

Considered at a higher level, these trends are not limited to just social media or just 
text. The media environment of the Internet is, in general, highly individualized. This is often 
a good thing. However, at the limit, we all end up atomized—unplugged from community 
and disconnected from the broader world. With AI this trend toward atomization will 
accelerate. Our information silos will shrink further, asymptotically approaching one-man 
bubbles. In the future, we might encounter the world primarily as a fantasy land of 


hallucinated, machine-generated content. 


For society to survive the coming flood of machine-generated content, there needs to be 
a countervailing force: technologies that can bolster collective sense-making and counter the 
gravitational pull of atomization. Otherwise, social reality will be so completely fractured 
that there won’t be any society left. As Alan Kay once said, “The best way to predict the 
future is to invent it.” Similarly, the best way to avoid a terrible future is to articulate a better 
alternative. 

There’s already an overwhelming amount of content online. The near-future of AI- 
generated content promises to greatly exacerbate this problem. Luckily, the same LLMs that 
power content generation can help sift through an increasingly noisy information ecosystem. 


However, this merely transforms a quantity problem into a quality problem. As we get more 
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information second-hand, filtered through LLMs, we must be more careful to discern the 
truth and context of what’s being conveyed. To survive and thrive in the era of Al-generated 
content, we need tools that leverage AI to help in all of these areas. It’s essential that we 
adopt tools that help us not only to get through more content, but also to validate claims and 


contextualize ideas. 


Solving the general case is a huge undertaking, but a new workspace for documents is a 
good place to start. I’ve started working on one. It lives, in its current form, at MagicPaper.ai 
—here you'll find the beginnings of an online workspace for distilling, summarizing, and 
correlating texts. The main insight is that the best way to validate texts and clarify context is 
by tracking the provenance of where ideas come from. Furthermore, when AI is leveraged to 
process information automatically, provenance can provide the missing link between the 


cursory view and full-length sources. Let me walk through some examples: 


Question Answering: First, load a document and then ask a question. The margins will fill 
with answers, each attached to a specific part of the document that backs it up. You then have 
the option to open the answers as a page of their own. Even once they’ve been lifted into their 


own document, the answers are all linked to the corresponding parts of the source. 


* The Kitchen Sink Conundrum and Simulation's Balancing x 


Answer x To simulate a world is almost a divine act. One Jewish tradition 
holds that there were multiple worlds created and destroyed before 
What is the kitchen sink our own. They didn't quite work out--beta versions perhaps?--and 
conundrum? so the universe was restarted again and again, until we got to the 


current version. 
Go 
But we will never simulate a world perfectly. First we must contend 
with the wrench of chaos theory--butterflies flapping and the 
inherent imprecision of measurement and all that--which has 
demonstrated that systems more complicated than a swinging 
Open pendulum can cause computational simulations to rapidly diverge 
over time despite small changes in the initial conditions. Start with a 
tiny rounding error, or a measurement mistake, and there's no 
guarantee that a prediction will be anywhere close to where the 
system will actually end up. 


ANSWERS 


A 7 The Kitchen Sink Conundrum refers to the tradeott 
But there is also something that might be called the Kitchen Sink between complexity and accuracy in computational 
Conundrum: as more and more detail--both in modeling features and mathematical models 

and data--is thrown into a simulation, there is no guarantee that it What is the kitcher 

will get us closer to a good understanding of reality itself (see also 


kitchen sink regression). 


Decades ago, in a RAND Corporation report from 1979 by David 
Leinweber, entitled "Models, Complexity, and Error," this was clearly 
articulated. This report examined two types of error: error of 
measurement and error of specification. 


Error of specification refers to how accurate the model is in 
accounting for the richness of the system being modeled. A more 
sophisticated model, with more operations on the input, will 
hopefully correspond better to the real world: it will be more 
accurate. So as the complexity of the model is increased, it will 
adhere better to reality and there will be a lower error of 
specification (though there may be diminishing returns). 


On the other hand, there is also error of measurement. The more 
complex a model, the more likely that any measurement error will 
compound, and cause the outputs to be wildly inaccurate. As 
Leinweber notes, "As the models grow larger and more complex, 
the compounded error in the prediction increases." 


So one curve goes down with complexity and the other goes up. 
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Extractive Summarization: Again, load a document into the tool. Click distill and two 
things will happen: fragments of the document will be automatically highlighted and you’ll 
be presented with a histogram. The histogram visualizes the distribution of fragment salience. 
You can use this chart to adjust the filter threshold. As you change the threshold, the 
highlights in the document will change. You can open the highlights as a page and they will 


remain linked to the source. 


~The Death of the Author x © Highlights of The Death of the Author x 


to attract an audience. Unsurprisingly, theorists and philosophers, — 


muru-gimensional space in wnicn & variety OF wrungs, none OF 


V Distill so often the harbingers of death, rarely gain mass appeal. Whether SENSE e A NN eae cae 
it's deconstructionists like Roland Barthes and Jacques Derrida, or Writing, he said, has taken on a new meaning. It is now apparent 
accelerationists like Nick Land, theorists get accused of _ that the author doesn't build a world of his own ex nihilo, but 
encouraging apocalypse. But that misses the point: their ideas rather channels innumerable cultural influences and presents not 
aren't aspirational, but despite this and perhaps because of this, a linear monolith, but rather a galaxy of signifiers, open to 
I they're often right. Camus put It well in his appraisal of Nietzsche. interpretation. This gives way to the Barthesian circle of life: "the 
mee ee Nietzsche, he said, “recognized nihilism for what it was an birth of the reader must be at the cost of the death of the 
examined it like a clinical fact." Theorists aren't in the business of Author." 
AUTO FILTER architecting utopia: their job is to diagnose the present. 
In the new world, prophesied by Barthes and others, and realized 
Returning to Barthes, his diagnosis was of course a prophesy of by technologists over the following decades, writers become 
FRAGMENTS Geathe The Dealt of tie Anton taken eh mere relay nodes in a global network of symbols and ideas. 
Hide - sea aE His ideas were buried by the explosive success of the Web, but 


twas 


already true, in 1977, that the genius of the author had been 
subsumed by semiotic bricolage. Accelerating technology rebuilt the 
literary environment to make this an unavoidable fact in the media 
we now inhabit. 


Crying out hopelessly from the zeitgeist of the '70s and ‘80s, a 
technologist named Ted Nelson wanted the media architects of his 
day to know that a profound transition had already begun, The 
author is now a bricoleur, he declared: our new media systems 
must honor this. As computing technology advanced and our media 
systems evolved, Ted remained a persistent advocate for a 
comprehensive redesign that he called Xanadu. 


among this set. | spent my senior year of college in close 
communication with Ted Nelson as | set out to write his story. The 
redesign of our media environment for its new life on computer 
screens felt like one of the most interesting chapters in the history 
of technology. And for me, as someone studying history, the fate of 
writing and research held a special significance. 
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then, decades later, many of his ideas were revived among a set 
of tech hipsters: roamers and digital gardeners and hypertext 
maximalists. 


When authorial genius becomes inaccessible, bright minds can 
turn instead to systematic curation and organization of ideas. 
This is what Ted sought: the writer of the future need not labor to 
produce a glimmering monolith of text, but rather, writers can 
enter into free play in the realm of networked ideas. 


Hypertext authorship centers a new kind of authorship, in which 
the network of ideas in the author's head can itself be shared, 
without being reduced to a line of words. The death of the author, 
however, will not give way to a new kind of author. The Barthesian 
circle of life will not be realized in the proliferation of a million 
hypertext authors. 


The text is where a “variety of writings, none of them original, 
blend and clash,” but until now these interactions couldn't be 
modeled comprehensively. A systematic way of reading will make 
the death of the author complete. At that point, when texts can 
be mathematically modeled as a galaxy of signifiers and the 
interactions between ideas can be calculated automatically, 
reading will finally outgrow its dependency on the author. 
Historically, it's been the job of the author to elaborate new ideas 
as an outgrowth and recombination of the ideas that came 
before. But soon, all these latent ideas will be readily 
discoverable by the reader and, empowered by new kinds of 
tools, readers will be free to roam farther than any author could 
reach. 


530 tokens _id:860316367-230 


Abstractive Summarization: This is the most complicated workflow, but also the one that I 
intend to focus on the most going forward. First you distill the key fragments from a 
document, then you generate a summary based on those fragments. Finally, you use another 
tool to correlate the summary with the input fragments. Now, every point in the summary will 


be connected to parts of the source text that support it. 


~ Summary of Hypertext x Hypertext (Wikipedia) x 


Hypertext is a technology that allows for the easy publication of 
information on the internet. It supports complex systems of ' ig 
linking and cross-referencing. The World Wide Web is the most He later worked with 
famous implementation of hypertext. The concept of hypertext Andries van Dam to develop the Hypertext Editing System (text 
was inspired by works such as "The Garden of Forking Paths" editing) in 1967 at Brown University. It was implemented using the 

i i ‘© terminal IBM 2250 with a light pen which was provided as a pointing 


ttre 


I Tim Berners-Lee proposed and © device. By 1976, its successor FRESS was used in a poetry class 
prototyped a new hypertext project in 1989, which led to the in which students could browse a hyperlinked set of poems and 


development of the World Wide Web. There have been various © discussion by experts, faculty and other students, in what was 
early implementations of hypertext, including FRESS, ZOG, and arguably the world's first online scholarly community which van 
HyperTies. Hypertext fiction has emerged as a unique style of © Dam says "foreshadowed wikis, blogs and communal documents of 
storytelling. Hypertext signifies a shift from linear to © all kinds". Ted Nelson said in the 1960s that he began 
decentralized forms of media. implementation of a hypertext system he theorized, which was 


named Project Xanadu, but his first and incomplete public release 
was finished much later, in 1998.Douglas Engelbart independently 
began working on his NLS system in 1962 at Stanford Research 
Institute, although delays in obtaining funding, personnel, and 
equipment meant that its key features were not completed until 
1968. In December of that year, Engelbart demonstrated a 
‘hypertext' (meaning editing) interface to the public for the first time, 
in what has come to be known as "The Mother of All Demos". 


ZOG, an early hypertext system, was developed at Carnegie Mellon 
University during the 1970s, used for documents on Nimitz class 
aircraft carriers, and later evolving as KMS (Knowledge 
Management System). 


The first hypermedia application is generally considered to be the 
Aspen Movie Map, implemented in 1978. The Movie Map allowed 
users to arbitrarily choose which way they wished to drive ina 
virtual cityscape, in two seasons (from actual photographs) as well 
as 3-D polygons. 


In 1980, Tim Berners-Lee created ENQUIRE, an early hypertext 
database system somewhat like a wiki but without hypertext 
punctuation, which was not invented until 1987. The early 1980s 
also saw a number of experimental "hyperediting" functions in word 
processors and hypermedia programs, many of whose features and 
terminology were later analogous to the World Wide Web. Guide, 
the first significant hypertext system for personal computers, was 
developed by Peter J. Brown at the University of Kent in 1982. 


In 1980, Roberto Busa, an Italian Jesuit priest and one of the 
166 tokens _id:2584359425-816 4,488 tokens _id:4144397038-240 
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Interestingly, a document workspace like above isn’t a new idea. It predates even the personal 
computer and the Web. Much of the design I’ve adopted in Magic Paper is inspired by ideas 
from decades ago. The provenance-tracking and parallel pages, specifically, are similar to 
designs Ted Nelson shared in Literary Machines (1981). These design primitives, I’ve found, 
are a perfectly countervailing force for the major challenges presented by AI adoption. 

By adopting a tool like Magic Paper, we can upgrade our information bandwidth to get 
through more content. This will be increasingly necessary as the volume of online content 
grows. Similarly, we'll need to put greater care into tracing ideas back to their source and 
context, as more of it comes second-hand through LLMs. 

The underlying designs here are nothing new. As we adopt AI in our knowledge work, 
we need to go back to the future of decades past. Advanced tools for reading and writing have 
been anticipated since the earliest days of personal computing, but they never really took off. 


Now, with LLMs, more powerful tools are both a possibility and a necessity. 
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The Fidelity of Text: Replicate or Rethink Text for 
VR? 


Michael Bonfert 


VR technology gives us a unique opportunity to reconsider how we approach text. The easy 
solution is replicating established text experiences, such as virtually simulating book pages 
and sticky notes or transferring flat web pages and PDFs to 3D space. This offers familiarity 
and ease of transition for users. Adhering to existing formats also accelerates technical 


realization. 


However, if we take the lazy route, we may not leverage the full potential of VR. I want 
to challenge the mere adoption of the status quo from physical documents or 2D screens. 
Instead, I advocate for a reinvention of how text can look and what it means in the boundless 
realms of VR. Questioning centuries-old conventions brings scary uncertainty and a 
temporary lack of standardization but could revolutionize how we approach knowledge, 


organize and convey thoughts, and collaborate in the long term. 


Take e-readers as an example with a smaller but similar opportunity. Thankfully, 
designers decided against merely displaying a page in its original formatting to represent the 
book experience as authentic as possible. Instead, e-readers reformat the content to fit the 


screen size and resolution with legible font size and useful augmentations. 


We refer to how closely users’ interactions with a VR system resemble a reference interaction 
(e.g., from the real world) as interaction fidelity. In a system with high fidelity regarding 
conventional text interactions, a user would perceive, experience, and manipulate VR text 
just as in the real world, for example, reading a virtual book and turning its pages. For some 
fidelity aspects, it certainly makes sense to ensure high-fidelity replications, such as for 
perceptual fidelity of the letters’ visual appearance, aligned closely with human sensory 
perceptions to ensure legibility and comfort in reading. But in which aspects should we aim 
deliberately for low fidelity? 


Compared to text representation on paper or screens, there are enormous affordances of 
the third dimension and the possibility to change the laws of physics in the virtual 
environment. What happens to text if it’s not restricted to a flat canvas? How could letters, 
glyphs, or symbols in 3D look like with an additional layer of meaning augmenting our flat 
alphabet? How could we organize documents or structure content without physical 
restrictions? How would it impact reading, writing, and editing a sentence, a chapter, or a 


library? What kind of multi-sensory experience could augment working with text? And 
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before you dismiss this as playful nonsense, remember that for many people, the feel of 
turning a page and the haptic feedback of a pen is essential to their engagement with text. We 
could explore entirely new forms of learning, data visualization, storytelling, collaboration, 
annotating, remembering, versioning, or concepts that I would not even find words to 


describe today. 


Instead of a linear sequence of characters, we could imagine text as three-dimensional 
structures, giving it materiality. It could help readers navigate or explore different chapters 
represented as spatial entities with connections and space in between. It could support fast 
reading or skimming with movement. It could directly incorporate multimodal elements or be 
integrated seamlessly into the virtual environment, giving it context, such as for learning or 
storytelling. Or it could be linked to the attention and focus of readers to foster and 
synchronize real-time collaboration. Spatial text can be dynamic, adaptable, tangible, rich in 
metadata, and a container for connected information. Let’s also make it highly adjustable to 
fit the user’s abilities and preferences. There are countless possibilities—but they need 
courageous dreaming and relentless testing. 

At the same time, we should be careful with altering the essence of text too much. 
Drastic changes can be fruitful but also compromise user engagement. To ensure practical, 
meaningful interactions with text, a user-centered approach is crucial for accessible, intuitive, 
and beneficial experiences. Finding the right balance and not losing touch with users’ 
expectations will be challenging. 

As we stand at the brink of a new era in virtual interaction, how we approach text in VR 
will significantly influence the adoption of the technology and impact our daily lives. Let’s 
step beyond the familiar and embrace the possibilities VR offers, reinventing text in a way 
that truly enhances our interaction with digital information. The concept of fidelity can help 
us guide this transformation as we reflect on which established forms of text should or should 


not be reproduced virtually. 
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The Open Protocol-ification of things 


Leon Van Kammen 


Online products rise and fall with the tides of popularity, while the open protocols that 
underpin them endure long after the products themselves have faded into obsolescence. 
Just as the internet itself emerged from the ashes of proprietary networks, so too do 
open protocols emerge from the ruins of once-dominant online products. 
The true legacy of online products lies in the enduring protocols they indirectly give 


birth to, which continue to shape the digital landscape for years to come. 
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Of Kings & God 


Frode Hegland 


< Reflections, perhaps ramblings, on where we are now, who we are, and the choice we have, 
written while sitting comfortably behind a nice Apple 27” Studio Display, listing to ZHU at 
Hakuba Iwatake in Nagano, Japan for Cercle, on YouTube, drinking a nice expresso on a bit 
of vanilla ice cream, before the bubble of XR really started to envelop our lives and at the 


very start of AI co-pilot authoring. Simpler days. > 


We did not have much choice in becoming. When our great, great ancestors moved 
from ocean to land, they did not have an opportunity to define who they wanted to become. 
When we moved from jungle to savannah, when we starting using tools, when we stood up, 
our ancestors did not have a chance to define who they should become. When we started 
constructing buildings, villages, cites and civilisations, we were carried along by progress 
entirely outside of any group’s control. 

Further, when we started developing new means of communication, starting with 
images on walls and moving onto images on clay, skin and paper, increasing the 
communication potential at every step of the day, we still had no agency as a species to 
decide who we wanted to be with these new media extending our minds and connecting us 


farther in time and space. 
This time it’s different. 


Now that we are about to, as it were, grab onto the frames of our monitors and the screens on 
our mobile phones and step inside—quite literally stepping into the rectangles on our phones 
and laptop computers—into fully immersive environments entirely created by us humans, we 
have an opportunity to be a part of the design of what this can be and more importantly, how 


and who we can be within them. 


King Charles II is said to have described St Paul’s Cathedral as awful and _artificial>®. 
The meaning of the words have changed and this is what is amusing; ‘Awful’ at the time 
meant awe-inspiring and ‘artificial’ meant designed and infused with art. Do we want to 
create our world of extended reality—cyberspace, metaverse, spatial computing, VR, AR— 
whatever we might want to call it to be awful and artificial in the modern sense or awe- 
inspiring and full of art? How do we want to human spirit to thrive in this new dimension of 


being? Or do we not want to thrive, only survive? 
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This time. We’re different. 


We now have a chance to define who we will be. We have the means to orchestrate large 
scale discussions and we have the technology to control what is build by the general 
community. We do not only need to purchase what the mega-companies decide is best for 
their financial positions. 


I’ve mentioned a King. I’d also like to bring God into this. 


God is not a narcissist. If you truly believe in a God, then take care of God’s creation 
and make it better for all God’s creatures. Don’t waste your time on hate for those who are 
different from you, worship God in a different way or under a different name. With our 
digital world expanding out beyond the rectangular computer screen, ask yourself how this 
new dimension of existence can be a holy place, a spiritual place, and not only a mechanical 
extension. If you do not believe in a God, please ask yourself the same question but phrase it 
as you might prefer. Ask yourself how this new dimension can extend what is good, not just 
for what can be done. 


Who we are. Now. 


Humans are an odd lot. On the one hand we launch a space telescope to study the deep 
history of the universe and on the other hand we deny that we are causing the climate on our 
own planet to change for the work. 


This comes from, at least in part, who we think we are and where we think we are 


going; our place in the world. 


In Restless Creatures [40] Matt Wilkinson writes how living creatures developed brains 
in order to move in their environments. In today’s political and socially tumultuous 
environment we may ask why so much politics is identity based. It strikes me as being related 
to movement-—in order to move in a defined manner we need to know where we are. 


Conducting ourselves. 


This is why we need to know our cultural and political identity in order to know how to 
move in the cultural and political worlds. A question then becomes; how can we build on this 


and re-define our identities to be more inclusive? 


I had the privilege of watching Beethoven’s 5th with my six year old son today on 
YouTube, beautifully conducted, with the music volume on loud. He asked me about the 
conductor and then it struck me that even though the Minority Report style information 
interaction is not sustainable simply because of the strain of holding up ones arms, we should 
be able to conduct our information in a similar way to how music is conducted. When we 
want to, not always. So far in XR there is a bit of a battle of interactions, with questions about 


whether controllers or hands are better and so on. When our use matures we will likely see 
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different interactions suiting different actions. Controllers in Quest and pinch moments while 
your hands are on your lap in visionOS is not enough. We will need much richer ways to be 
ourselves in XR. 

Some people who are especially proficient with software look like they are almost 
working a musical instrument when they work. Doug Engelbart was described as dealing 
lightning with both hands?’ during is seminal 1968 demo. 

These two thoughts come together with the simple dream of people bigger in the world. Can 
we use extended reality to both allow our sense of who we are to be bigger—more inclusive— 
and can we act bigger in this world, conduct our affairs as richly as musical conductors or 
dancers? 

XR: Best & worst. 

At the worst and most dystopian, XR can become a prison cell for the mind. 

At the best, XR can unleash the full human body to express and comprehend like 
nothing we have experienced before. 

How can we unleash the mind expanding power of XR? When I asked Jimmy Wales 
how he managed to create a system whereby Wikipedia does not simply devolve into chaos, 
he simply replied that they have been working on making it easy to do good things, and hard 
to do bad things. The definition of what is good and bad may differ for some people but boy, 
what a design principle! 

The question is moral and practical and practical equals democratic: The more practical 
it is to carry out a task, the more likely more people can do it. 

On the moral side, can we gather together to write an extended version of the 
Declaration Of Independence for Cyberspace?3® What would you like to see in such a 
statement? Who should write it, and who should edit it? 


The same goes for AI. 
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This is what I think we should do. 


But first, if you agree or disagree, can we connect our thoughts? Can you write 
yours down? 
The power of writing down our thoughts, the power of text to help clarify our thoughts and to 
communicate them with others—is unparalleled. And useful for reflecting on the future. Great 
thinkers used to communicate via hand written letters. Now we have instant communication 
available but we choose to put much of our thoughts into social networks where what we 
have written slowly fades from being accessible with a bit of scrolling to being accessible 
only through searching. I feel it’s not enough to ‘write it down’ in a Tweet or similar where it 


melts into the ether. Write a document and connect it to this one. You can cite me on this. 


I think we should: 

* Broaden the dialog of what the future of text can be—particularly in XR and with AI as co- 
pilot ally and competing adversary—with a wider community. 

¢ Experiment to experience different ways of working in XR. 

¢ Focus on making user data and metadata—the data which makes the data useful—truly open 
and available so that users can take it with them. 

And this is exactly what I am doing. But our community at this point is tiny and 
needs to grow, as well as entirely different communities should grow and address the issues 
in different ways, allowing us compete with ideas and learn from each other, what Doug 
Engelbart would have called Networked Improvement Communities (NIC). Are you doing 
something similar? Join us? Are you working on something different or do you disagree with 
our approach but you still think it’s time our species exercises some degree of control of its 


evolution? Then join us in a network. 


Interaction 


Our information, our knowledge and how we use it—how we can interact with it-is who 


we are. We are our thoughts and our actions. 
The way we interact with our thoughts is how we think. 


When we externalise our thoughts with physical or digital media to think we are 
externalising parts of who we are. The way we can interact with our externalised thoughts 


shapes who we are, how we think. 


We are what our interfaces allow us to be. 
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Information 

We used to own our information, when it was on paper. Now that it is digital some of it 
we can’t take out of one virtual folder and into another. We used to have the same 
information in a manuscript as we did in the final published work. Now that we have greater 
interactivity with our manuscripts we strip them of much useful information in order to 
conform to paper-based expectations. 

There will be cases where this will not be practical, such as for highly optimised 
modelling of various sorts. For the connective tissue, the addressing (links) and packaging 
(documents) of knowledge however, there should be great freedom for anyone to develop 
interactions. 

We need to ask. Who will we be in this world and who will be the creators? Will we let 
capitalism define the entire environment through large companies or will we manage to 
someone connect, using the most powerful communications technologies the world has ever 
seen, to develop systems and infrastructures to support truly open development? 

To be clear, this is not to say I think all software should be free and open. 


To be clear, this is to say that information should be owned by the user and transferable 


and that infrastructures should be open in all sense of the word. 


To be clear, this is to say we can, and we must strive for wide and inclusive dialog as to 


how to achieve this and what ‘this’ should really mean. 


We need to ask whether we will slap headsets on our cranium to block out view of 
the world, or to amplify our understanding of it—our place in it—and how we relate to 


others in it. 


< end rambling. Onto practical realities > 
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Rich Information Environments 


Frode Hegland 


Design thoughts on space and time at the close of 2023. 


2D & 3D 


Being able to include graphs/charts/maps which are not static but can be interacted with by 
the reader and extracted as data will be powerfully useful. Issues around such live graphs will 
abound and we are sure to learn more issues as we develop systems to enable them but we 
can already consider how to interact with them, in context of traditional digital screens and 


interactions in XR environments. 


A key issue in XR will be how to deal with large graphs and charts, and one thing will 
mean efficient hiding and selecting, which we have now in the Map view in Author, as well 
as how connect graphs. To make the hiding and selecting more transparent to the user we 
need a ctrl-click menu in Maps, which is why I am so keen on this. Can you do this or would 
it require a large amount of change? 

Related is the issue of nesting. How do you think we should make this possible? 
Maybe something as simple as being able to either drag a .liquid document onto the Map and 
it appears as the document title (from the Export dialog which would be the name in Visual- 
Meta) or name if no title has been assigned yet? If so, it should appear with [hard brackets] to 
indicate it’s a link. Click and it will act as any link, with the ‘delete’ or ‘jump’ option. 

Another issue is with Visual-Meta for graphs/map to be in PDF’s. For this, can you 
simply add the JSON from the .liquid file, with a sentence description in the Visual-Meta 
description header? Or is there a better way for Reader macOS and Reader in web XR to 
parse this? Brandel, who works for Apple on visionOS has already done a test on this, which 
only works in Chrome: Take a .liquid document with a Map in it and drop it onto this URL 
and it will be viewable in XR (Quest etc.): https://t.co/nEloUpiUsW 
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AD : Events & Timelines 


In the Future Text Lab we have been discussing timelines and this is of course a tricky 
subject. Not just because time itself is tricky [41] but because of the way us humans refer to 
time and how we think about time. We say things like “ in half an hour” and “next Tuesday 
afternoon” as well as “the thirteen hundreds”. Apart from very specific events there is a large 


amount of fuzziness around most of the way we communicate time. 


We having been discussing formats and interchanges to deal with this but a test of 
selecting a range of timeline entries from this book and asking ChatGPT to format it as .ics 
(iCal format) actually worked surprisingly well. There were issues with the fuzziness, but the 
AI did the job. We should therefore probably look at AI workflows. 


AI workflows. There is no question that even current AI systems can provide powerful 
analysis and transforms. An issue is that currently they exist as separate systems for pasting 
in data and prompts, or are embedded for special use in applications. A useful approach could 
be to look at the workflows or pipelines for AI; How can a user take a document (a PDF, a 
spreadsheet, a calendar entry etc.) and add it to a timeline app for the data/dates to appear and 
be as interactive as they wish? I can imagine a workflow with my own tools where the user 
selects text in Author or Reader and uses AI to convert into a useful data format, perhaps 
iCal/.ics and shows the results to the user for verification in a simple table where the user can 
easily delete any mistakes. This can then be copied or sent to other software for viewing, as 
well as automatically be added as a temporary AI additional Visual-Meta Appendix. We ’re 


working on this 
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User Fuzzy. For this to be possible the events need to be encoded in the document in a robust 
and accurate way, including the level of fuzziness. A question then becomes who should 
specify the level of fuzziness; the author or the reader, or perhaps both? Might this be similar 
to how text is dealt with, where the author chooses a typeface and the reader can sometimes 


want to change it? Might fuzziness be part of the typeface of time? 


Picture a settings dialog, as the one mocked up below, where there are default seeings 


for fuzziness but the user can change the level. 


Carlo Rovelli writes in The Order of Time [42]: “There is no need in any of this to choose a 
privileged variable and call it ‘time’. What we need, if we want to do science, is a theory that 
tells us how the variables change with respect to each other. That is to say, how one changes 
when others change. The fundamental theory of the world must be constructed in this way; it 
does not need a time variable: it needs to tell us only how the things that we see in the world 
vary with respect to each other. That is to say, what the relations may be between these 
variables.” He goes on to explain that “there is nothing mysterious about the absence of time 
in the fundamental equation of quantum gravity. It is only the consequence of the fact that, at 
the fundamental level, no special variable exists. The theory does not describe how things 
evolve in time. The theory describes how things change one in respect to the others, how 
things happen in the world in relation to each other. That's all there is to it.” This can inform 
how we choose to record time and visualise time by prioritising events rather than an idea of 


a timeline to which all must connect. 
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Then also picture a timeline with zero fuzziness, where scales are shown based on the 


timescale presented, such as below, where years are added sequentially with no gaps for years 


with no entries, and century marks are shown as years also vertically, much like a heading. As 


long as there is good data and user settings, flexible views, based on themes perhaps, can ben 


generated: 


1700 


1702 The Daily Courant, the world’s first daily newspaper, printed on paper so cheap it was designed to be thrown away after reading 
1704 Daniel Defoe, considered the first journalist, publishes The Review 

1704 Newton's Opticks, the first major scientific book published in English, not Latin 

1706 Newton's Opticks translated into Latin 

1714 First patent for a mechanical typewriter issued to Henry Mill 

1723 De Etruria regali libri VII Thomas Dempster used sans serif typeface to represent inscriptions in Ancient Greek and Etruscan 

1725 Improvement to the Jacquard-type loom by Basile Bouchon who introduced the principle of using a perforated band of paper 

1731 First peer-reviewed journal, Medical Essays and Observations (Philosophical Society of Edinburgh, Edinburgh). 

1739 Last international treaty written in Latin, the Treaty of Belgrade, indicating the new pre-eminence of living languages over dead ones 
1748 First modem use of sans-serif (“grotesque”) lettering, anonymous letter carver, grotto at Stourhead, England 

1755 A Dictionary of the English Language by Samuel Johnson 

1767 Index Card organization by Carl Linnaeus 

1769 Every house in Britain needs to have a number for addressing, introduced with the Stamp Act, 

1770 Natural rubber used as an eraser by Edward Nairne 

1771 UK Parliament formally gives journalists the right to report proceedings 

1772 Aquatint printing by Peter Perez Burdett, named by Paul Sandby 

1780 Didot and Bodoni by Firmin Didot and Giambattista Bodoni, the first modern modern Roman typefaces 

1780 First card catalog by librarian Gottfried van Swieten, Prefect of the Imperial Library, Austria 

1783 James Madison of Virginia proposes the creation of a congressional library 

1786 Rounded sans-serif script font developed by Valentin Haily for the use of the blind to read with their fingers 

1787 Constitution of the United States, mentioned here as a milestone in written documents producing and framing a society 

1787 The Federalist Papers by Alexander Hamilton with John Jay and James Madison in The Independent Journal, considered the most important documents for interpreting and 
understanding the original intent of the Constitution of the United States 

1791 First card catalog for libraries, using the back of playing cards by a group of men with bibliographic experience led by Barthélemy Mercier 
1795 Modem Pencil by Nicholas-Jacques Conté 

1796 Lithography by Alois Senefelder 

1796 Colour Lithography by Alois Senefelder 

1799 The Fourdrinier machine, a continuous paper making machine by Louis-Nicolas Robert of France 


No matter how it is displayed the interactivity will be key. For example, in this book you can 


select any text and do cmd-F and if the text is in our timeline it will appear, helping you 


access the timeline from anywhere in this document. 
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Result of Dialog with Leon Van Kammen 


Leon and I had two meetings to discuss timelines but we ended up going all over the place 


and spending a good amount looking at the usefulness of chatGPT in Reader. 
https://youtu.be/EeZhU-Y_Ss4 
https://youtu.be/TSRkGF30SFO 


Multi-query workflow 

Our playing around revealed that even the best prompts we could make, such as “Take this 
text and extract all names, then use your knowledge to list chronologically by birth” did not 
complete with only one result. We kept getting results which included both sections or only 
one. We decided to put together a mockup of an improvement to the current dialog in Reader 
where the user can make their own custom prompts to use on selected text. The addition is 
the ability to add prompts, where each new prompt will act on the result of the previous. This 
can allow for much more sophisticated and useful work to be done. Furthermore, the results 


should be shown in an editable window: 


Title: “Show people Chronologically 


Preface: Extract names of people in this text. 


Cancel Ok 
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Store results as Appendix 


The second thing is an option for the resulting AI work to be saved in an appendix at the end 
of the document for instant access should the user call the same command again. This would 
be in the Visual-Meta style and would be clearly marked as an AI appendix and would 
automatically be removed on ‘forcing’ this command to be done again (or if the user simply 


wants to remove it). A ‘Save Results’ button in the results screen has been mocked up here: 


eee Names Chronologically 


List of names sorted by creation dates: 

- Aeneas (mythical origins of Rome, third century B. C.) 
- Quintus Fabius Pictor (second century B. C.) 

- Lucius Cincius Alimentus (second century B. C.) 
- Polybius (second century B. C.) 

- Diodorus the Sicilian (30 B. C.) 

- Marcus Porcius Cato (around 168 B. C.) 

- Sulla (first century B. C.) 

- Valerius Antias (first century B. C.) 

- Marcus Terentius Varro (first century B. C.) 

- Livy (27 B. C. to 19 A. D.) 


- Dionysius of Halicarnassus (first century B. C.) 


Save Results 


These are the thoughts as they are developed today, as to how to use AI in a workflow, not 
just as a chat area of to the side or simply embedded in an app. In the results above, the 


results can easily be copied for other uses, as it can be done today. 
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Reflections in Time 
Peter J. Wasilko 


In recent days our Future Text Lab has been taking up the topic of timelines and temporal 


visualizations. 


Linear Time 


Direct uniform visualization of long spans can be quite challenging depending on our domain 
of interest, the sheer volume of data we want to plot, and the uniformity of scale of any spans 
of time that we need to represent. Are we including both Aeons of Geological Time and the 
durations of Geological Research Expeditions? A simple PRINTED timeline only works if 
that data can be represented on a conventional page (or series of pages) at a visual scale that 
permits the data to be discerned. To save space the timeline can be segmented and stacked, 


but searching for events whose date isn’t already known is problematic at best. 


Elastic Time 


In VR we can embed the timeline in a zoomable user interface allowing us to dynamically 
play with scale and zoom in on periods of interest to whatever granularity is necessary to 


resolve the smallest temporal units in our record. 


Nested Time 


In PRINT it is often useful to nest multiple timescales. This is the genius of a Calendar’s 2-D 
layout, with a Year represented as twelve sub-grids to record its Months, which are 
themselves decomposed into stacked Weeks and Individual Days with an optional list of 
events, perhaps with a cross-reference to a Day Planner possibly reflecting Day & Night and 
Weekend/Workday distinctions with background shading and stacking the hours and nested 


minutes of each day with space to write in events of interest. 


In VR we can nest more deeply in a single visualization zooming fluidly from centuries 


to micro-seconds in 1-D, 2-D, and 3-D representations. 
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Elided Time 


Where our timeline entries are Sparse as in the Dates of World’s Fairs in the 20th Century or 
bursts of chat activity between parties of interest we can Elide or Fold away the periods of 


inactivity leaving behind a marker like “Three Days Later”. 


Clustered Time 


Where bursts of activity are both numerous and dense, say dozens of communications within 
a span of minutes punctuated by gaps at different scales between hours and weeks it makes 


sense to merge individual records into Centroids representing each session of interest. 


In VR we can Nest and Unfold such clusters. 


Parallel Time 


When we find that the events under consideration begin to overlap, the most common 
solution is to spread them out into separate parallel Bands. This can be based on Event Type 
as in Conferences vs Expeditions vs Discoveries or by Actor or Actor Types as in a Doug 
Engelbart timeline running in parallel with a Ted Nelson timeline, perhaps with timeline 
Adornments added to label any events in which they both participated. 

The MIT Media Lab’s Chat Circles transcript user interface used this approach with 
parallel vertical timelines representing each speaker, punctuated by horizontal bars whose 
position and width reflected the time and length of each utterance which could be displayed 


upon mousing over each indicator bar. 


Synchronized Time 


If multiple timescales are involved, we can stack Multiple Timelines and synchronized 
scrollbar Controllers, each representing a different scale of interest with its thumbnail sized to 
reflect the relative percentage of the visible region of the controlled timeline at that scale, so 
the two weeks in focus spanned by a conference might be a tick mark on the Century scroller, 
a narrow bar on the Years scroller, and progressively wider bars on the Months and Days 


scrollers, whilst the scrollers representing shorter durations would be disabled. 


In VR can we fuse these different scales into a navigable volume so we can fluidly scrub 


through time at different scales and rates by waving a hand in the control volume. 
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Aligned Event Time 


In many applications the absolute start dates of events are irrelevant and what we actually 


care about is the durations between members of event clusters. 


The canonical example here would be Medical Diagnostics where we might have 
several hundred data records of the progression of an illness from first manifestation to final 
outcome. Plotting them on an absolute timeline will scatter them over the duration of the 


study making it nearly impossible to discern patterns of interest. 


The solution is to Align all of the records by their start dates and then plot all 
subsequent dates relative there to. The University of Maryland at College Park’s Human 


Computer Interaction Lab has explored this approach. 


Higher Order Time 


In other cases we might not want to focus on individual events, but rather on their frequency 
and the duration of the spans between event clusters. This can be done by looking at Event 
Counts Per Unit Time or Durations Between Events and the Change In Event Counts Per Unit 
Time or the Change in Duration between Events Over Time, which is to say that we want to 
take a Derivative of the rate at which the frequency of events is Speeding Up or Slowing 


Down. 


This approach allows us to then align these higher order patterns and look for 
correlations with triggering events, such as the tendency of calling someone a Nazi or asking 


for money to end a conversation. 


Conclusion 


As we have seen there are a number of ways in which we can visualize time and event 
streams, whose suitability of use will vary depending on the data at hand and what we want 
to do with it. None is inherently superior, each has its uses, and only through mixing and 
matching them on a case by case basis can such affordances achieve their full potential. 
Unlike traditional Print media or even the dramatically more flexible 2-D displays in 
common daily use, VR gives us the added measure of flexibility to fuse all of these ideas 


into exciting new Temporal Interfaces of the Future! 
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Cognitive productivity, consciousness and 


hypertext 


Luc P. Beaudoin & Rachel Horst 


Introduction 


In this paper we address the question of whether and how hypertext can support scaffold 
consciousness to enhance cognitive productivity. This calls for a cognitive productivity?? 
framework and a theory of consciousness. Here, we rely on Beaudoin’s (2014, 2018) 
cognitive productivity framework and Merlin Donald’s Multiple Component Convergence 


theory of consciousness. 


Cognitive Productivity Framework 


Cognitive productivity transcends basic productivity. Beaudoin*® defines cognitive 
productivity as the efficiency with which one uses information to solve problems, build 


products (including conceptual artifacts), and improves oneself. 


Input Output 


iiomaton —> RSS) 


/\ 
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Figure 1. Uses of information (from Beaudoin, 2013*') 
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He defines meta-effectiveness as the skills, dispositions and manifold underlying 


information-processing mechanisms [mindware] that enable and drive people to improve 


themselves.” 


The Cognitive Productivity framework, illustrated in Figure 2 below, proffers seven 


principles for getting smarter with knowledge and technology, organized in three categories 


as follows: 


I. Principles of self-governance 


I. 


Principle 1: Lead yourself with knowledge. Improving oneself, and excelling, with 
knowledge is an effortful process that requires personal leadership. This requires seeing 
oneself as an information processor; carefully selecting information sources; and directing 
one’s information-processing towards one’s values, mediated by one’s projects as 
illustrated in Figure 2.1 below. 

Principle 2: Manage your cognitive life mindfully. Three areas of management are 


singled out in the framework: projects, knowledge sources and attention. 


Principles of productive information-processing 


Principle 3: Assess analytically. Assessing information is more difficult than it may seem. 
The CRAAP criteria are often taught to students, who are advised to assess the Currency 
of the information, the Reliability of information, the Authority of the author, the 
Accuracy of the information, and the purpose of the author. These are good starting points; 
however, the Cognitive Productivity framework provides a more systematic assessment 


schema for advanced undergraduates and knowledge workers, briefly described below. 


Principle 4. Surf strategically. Surfing refers to all the superficial but often necessary 
processing of information such as discovering new documents, skimming documents, 
bookmarking, accessing information, creating PDFs, and creating tables of contents. 
Principle 5. Delve deeply. Delving is the process of deeply processing a knowledge 
resource. Some academics call this “constructively responsive reading”, mean- ing that 
you respond to a text by constructing your interpretations, evaluations, and so forth. In his 
famous book, How to Read a Book, which one of us reviewed in 2014, Adler calls delving 
“analytical reading”. But, of course, in modern times we are concerned with much more 
than text; there are podcasts, audio books, videos, and more. The Cognitive Productivity 
framework suggests cognitively productive ways of delving with such software and using 
note-taking software. 
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III. Principles of mastery 

* Principle 6. Practice productively. Psychological literature in education and expertise has 
clearly and repeatedly demonstrated that practicing is essential for peak performance. Yet 
few knowledge workers admit to rehearsing information. This is largely due to a number 
of illusions that are documented in section 3.3.2 of Cognitive Productivity: Using 
Knowledge to Become Profoundly Effective. For instance, because reading comes so 
naturally for them, and because they don’t feel they will be explicitly tested (and graded), 
knowledge workers often wrongly assume they will be able to remember, understand, and 
apply knowledge gems from articles, books, seminars, podcasts, youtube videos, and other 
media they process. Productive practice is a framework for practicing, based on empirical 
psychology of test-enhanced learning, deliberate practice, memory retrieval, and more. 
Flashcard software like Anki and Remnote can be coopted for productive practice. 

* Principle 7. Apply knowledge. What is the point of delving if one can never recognize 
when previously delved information currently applies, nor apply the knowledge? 


Strategies and software exist to facilitate the personal transfer (usage) of information. 


These principles are depicted in Figure 2 below. 


Surf Strategically(4 
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Sources 
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Figure 2. Seven Principles for Using Knowledge to Become Even More Effective (from 
Beaudoin, 2018) 
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Figure 2.1 Leading oneself with knowledge 


Merlin Donald’s Merlin Donald's Multi-component Convergent theory of consciousness 


Before well into the cognitive revolution, consciousness was a taboo subject considered non- 
scientific amongst empirical psychologists. Many hard-nosed neuroscientists still feel that 
way. In Nature Reviews Neuroscience, Anil Seth and Tim Bayne reviewed several theories*?. 
Unfortunately, they left out one of the most compelling theories, Merlin Donald’s Multiple 
Component Convergence (“MCC”) theory of consciousness described in his A Mind So 
Rare* book and other publications. This is the theory of consciousness that has focused the 
most on computing technology. In our view, this provides a basis for understanding 
consciousness that is analogous in importance to the basis for understanding evolution that 
The Origin of Species provided for evolution. 

Merlin’s theory emphasizes three major transitions in the evolution of human 
consciousness. First, there was the memetic transition, in which humans became able to use 
gesture, mime and imitation. They also then became able to rapidly compile skills. Second, 
there was the mythic transition in which humans became able to tell, understand, repeat and 
apply stories. Third, and only very recently, came the theoretic transition in which humans 
were able to escape the limitations of the biological brain’s internal symbol systems by 
creating, decoding and understanding external symbolic representations. Writing systems and 
computer systems become tremendous cognitive assets astronomically extending the human 
mind-brain. Whereas most theories of consciousness nearly identify consciousness with 
sensory binding and working memory, Donald’s theory emphasizes, in addition to these 


shorter time frames of information integration, intermediate and long-term governance 
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operating over time frames of (say) a conversation and much of a lifetime. Human 
consciousness makes use of something akin to what K. Anders Ericsson called long-term 
working-memory [43] (discussed in Beaudoin’s first Cognitive Productivity*+ book and a 
paper by Peter F. Delaney*>), which Donald’s theory entails is not reserved to expertise in the 
technical sense of the word but evinced by all normal humans. Human consciousness does 
not require 10,000 hours of domain-specific practice to be able to rapidly encode and decode 
information in a domain, given human ability to use what Donald in pp. 308-326 of a Mind 
So Rare refers to as an “external memory field” (such as books and papers, the content of 
which can rapidly be loaded into working memory). 

Luc P. Beaudoin asked Merlin Donald what his theory should be called. In an email 
dated December 31, 2022 Merlin Donald responded to Beaudoin as follows (emphasis added 
by us): 


“T think my theory of consciousness is functional — I dismissed the “hard problem” as 
currently unsolvable and focused on the “easy” part (!) — namely, the identifiable 
evolutionary origins of our capacity for conscious integration. The brain mechanisms are not 
yet clear but are at least open to investigation. 

It is also best described as a multi-component theory. I proposed that our capacity for the 
conscious integration of multiple sources of knowledge came into being as vertebrate species 
evolved different capacities — often in different species — for perceptual binding, short-term 
memory, attention control, and meta-cognition, among other things, each with its own 
evolutionary trajectory, which happened to converge, for its own adaptive reasons, at a 


certain level only in Homo sapiens. 


So, much of our conscious capacity has a very ancient and deep origin in brain evolution. The 
uniquely human part of our capacity also relies heavily on a capacity for “slow” processing, 
or wide temporal integration, which is uniquely evolved in humanity, and enables us to 
compress a very temporally wide event — a story, for example, or a hockey game, or a long 
hunt — into a chunk or memory unit, instead of a series of complex events. We have no idea 
how this is done by the nervous system, but it obviously happens, and does not depend 
entirely upon language. 

Most importantly, it is also a “scaffolding” theory: that is, our basic brain-based systems 
have been extended by the use of cognitive tools of various kinds, as scaffolding devices, 
which magnifies the evolutionary distance we have gained on fellow primates, by orders of 
magnitude. This includes our pragmatic social use (in idea-laundering systems) of other 
peoples’ memory systems, the tremendous gains in our retrieval options afforded by the 


invention of languages, and various crucial cognitive technologies including writing and 
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various electronic devices. 


So, what to call anything so complex? A good question! The key to our cognitive success is 


the convergence of so many independent evolutionary trajectories in our species, so 


perhaps it is best identified as a “convergence” theory of consciousness, or a “multiple 


component convergence” theory.” 


MCC is also distinctive from other theories in emphasizing the role of culture (as 


explanandum and explanans) and the functional architecture of consciousness. There’s much 


more to the MCC theory than can be described in space available here, as is suggested by 


Figure 3 below. 


Some concepts in Merlin Donald’s Multiple-Component Convergence theory of consciousness 
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ai 
1. Memetic (“GISM”: Gesture, Imitation, Skill, Mime) 
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Inputs: all secondary cortical regions, etc. 
Outputs: secondary motor regions, 
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cultural materialism, the great hominid escape from 
the nervous system; symbolic technology, external 
memory fields, etc. 


Figure 3. Some MCC concepts 
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Evolutionary 
Scaffolding 


* Subsymbolic 


Scaffolding consciousness for cognitive productivity 


Having briefly sketched a framework of technology-enabled cognitive productivity and the 
MCC theory of consciousness, we can now briefly address some of the ways in which 
hypertext software and strategies can scaffold MCC-conceived consciousness and support the 


seven principles of cognitive productivity*®. 


I. Principles of self-governance 

¢ Principle 1: Lead yourself with knowledge. Here one can make a list of the principles of 
cognitive productivity themselves, and commit them to memory using flashcard software 
such as Anki. One can also clarify, assess and redefine one’s values and one’s projects. 
These are examples of long-term consciousness. 

* Principle 2: Manage your cognitive life mindfully. To manage one’s projects, one can 
select personal project management software. For all the principles, we recommend using 
software that conforms to the manifesto for ubiquitous hyperlinking*’, for reasons 
explained in principle 5, surf strategically. OmniFocus and Things are good examples; 
ToDoist in contrast lacks an API for hyperlinking. To manage one’s knowledge sources 
one needs a system. The highest caliber information produced by humans is normally 
contained in PDFs. There are many types of apps to manage PDFs, such as bibliography 
apps (e.g., Zotero and Bookends), and general information managers (such as 
DEVONthink* and EagleFiler*?). This too illustrates temporally expansive 
consciousness, long-term governance. Managing one’s attention is a critical way to 
manage consciousness; attention itself is a component of consciousness. Here it helps to 
use time-tracking software such as Timing*®, which is linkable. mySelfQuantifier?! is a 
free spreadsheet-based time-tracking system developed by the first author and based on 


cognitive science. 


II. Principles of productive information-processing 

¢ Principle 3: Assess analytically. The Cognitive Productivity framework contains the 
CUP’A schema for systematically assessing knowledge resources. This framework was 
discussed in the 2021 edition of the Future of Text book [44]. CUP’A is a pragmatic, 
detailed technology-enabled schema pertaining to the overall helpfulness of information. 
Information can be assessed with respect to its objective Caliber, Usefulness to one’s 
projects, Potency (potential to alter oneself), and Appeal to one’s attitudes (preferences). 
Each of these criteria is complex. The latter is a potential red herring (information can be 


seductively appealing yet irrelevant or counterproductive). Using an external notes 
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document that is bidirectionally linked to the source is a powerful way to assess a 
document. 

Principle 4. Surf strategically. While ‘surfing’ refers to all manner of superficial 
information processing, it is nonetheless critical to cognitive productivity. An analogy can 
be made with plumbing in a house: it’s not lofty, but necessary. Particularly relevant to 
consciousness is the 2-s rule of surfing, according to which each of 80% of the 
information resources one accesses should take no more than 2s to access. Searching 
(whether web searches or with desktop search tools like Spotlight®) requires more than 
2s. Implementing the 2s rule requires contextual information software which essentially 
extends consciousness, emulating long-term working memory. Such software (like the 
Hookmark*? app for macOS) enables users to access information related to their current 
focus. For instance, while writing a paper, one can access a ‘hooked’ (bidirectionally 
linked) outline, emails from the editor, a task list, and more, as illustrated in this video>?. 
This protects working memory‘ and psychological flow>>. Similarly, one can use such 
software to bidirectionally link and access notes pertaining to a source. Contextual 
information retrieval requires apps that are linkable through automation, as described in 
the Manifesto for Ubiquitous Linking>® discussed in the 2021 edition of The Future of 
Text. See also Beaudoin (2023) How contextual information-retrieval software can 
support knowledge-building by extending intermediate and long-term consciousness [45]. 
Principle 5. Delve deeply. It is easier to delve deeply if one uses software with hypertext 
features that support delving, such as Reader?’ by The Augmented Text Company and the 
ironically named PDF reader, Skim>’. This applies also to web browsers, ebook readers, 
podcasting apps, etc. Software for delving hypertext should enable users not merely to 
highlight text with different colors, but to categorize text using an educationally useful 
taxonomy. For instance, one should be able to tag text as being important, something one 
does not yet understand, something one disagrees with, or an ancillary claim. The figure 
below illustrates how one can map colors to tags, but (again) ideally one should be able to 
apply qualitative tags. 
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One should then be able to filter the the source text and one’s annotations according to the 


category. This is a form of information retrieval (contextual information retrieval being 


essential to consciousness in MCC) that enables consciousness to be loaded with 


pedagogically relevant information. This facility provides many benefits. For instance, 


students who know what they don’t understand are better equipped to fix the flaws in their 


knowledge. Tagging text as “don’t understand”, even for an expert, can yield significant 


cognitive benefits. 


III. Principles of mastery. 


Principle 6. Practice productively. MCC emphasizes review and rehearsal as capabilities 
that distinguish humans from other animals. No other animal reviews or rehearses on the 
same scale as humans, and the benefits of these practices are enormous. It is commonly 
but falsely believed that flashcard based learning is merely for rote learning of facts. See 
for instance, Jeffrey D Karpicke 1, and Janell R Blunt’s “Retrieval practice produces more 
learning than elaborative studying with concept mapping”°’. Cognitive Productivity 
books have developed a form of practice that generalizes flashcard learning and yet can be 
performed with advanced flashcard software such as Anki. Instead of merely having two- 
sided cards, Anki supports the definition of template-based notes with multiple 
configurable fields, enabling users to define data types. This supports learning names, 
procedures, terms and condition-action rules. The difference between Anki and basic 
flashcard software is analogous to the difference between an advanced programming 
language such as Swift and a low-level programming language such as assembler. The 
goal of productive practice is to emulate having a coach. Deliberate practice, as noted, can 


scaffold consciousness by enabling learners to rapidly load information into working 
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memory with external or internal cues. 

* Principle 7. Apply knowledge. The ‘memory transfer problem’ has been described as the 
biggest problem in psychology of memory. This transfer problem is the problem of 
recognizing what previously learnt knowledge is applicable to the current situation. The 
first hurdle to overcome is understanding the problem of transfer and its importance. This 
can best be achieved by getting into the habit of developing instillers (containing 
flashcards, per principle 6) and testing oneself on one’s previous reading. One might select 
a random book or paper that one has read and ask someone to test you on it. Ask oneself, 
when’s the last time one’s used the information? One might realize that much knowledge 
lies fallow. Productive practice (Principle 6) is meant to help one with the problems of 
applying knowledge. But other practices are required. Regularly reviewing information 
helps ensure it will be used. Reading clubs are potentially useful; however, they need to be 
restructured to deal with problems of applying knowledge. Motivation to apply knowledge 
needs to be enhanced by using the best motivation theories (e.g., the status signaling 


theory [46] of Will Storr and Tori Higgins’ regulatory focus theory®!), 


Conclusion 


We have illustrated how cognitive productivity can be enhanced with hypertext mediated by 
consciousness. There is no consensual theory of consciousness. We chose Merlin Donald’s 
Multiple Component Convergence theory of consciousness because we find it the most 
compelling, and this is the theory that has focused the most on technology. For a more 
elaborate presentation of this argument, we refer you to Beaudoin, L.P. (2013) How 
contextual information-retrieval software can support knowledge-building by extending 


intermediate and long-term consciousness [47] and Cognitive Productivity® books. 
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In Closing : On Virtual and Augmented Reality 


Vint Cerf 


VR and AR have been the ‘coming technology’ for some time now. 

A major challenge has always been comfort or discomfort of wearing a headset for long 
periods of time. For headsets to work in AR, RR (real reality) needs to come through and be 
visible in high resolution in addition to any augmentation. Interactions in AR/VR have also 
been challenging. Simulated hands and tools have been made to work but without haptic 
feedback, they are less intuitive. 

Holographic displays would be a potential way around the usual headset but often have 
limited applicability for groups since the focus may only be optimal for parties at a particular 
focal point. 

Direct manipulation is more intuitive and attractive when this is possible. As we have 
learned with chatbots, application programming interfaces that allow the bot to interact with 
the real world through the virtual one is empowering. This could extend to direct control over 
‘Internet of Things’ devices and general software applications. 

VR environments often need avatars to facilitate interaction among participants. 

One wonders to what extent avatars should resemble the real users? Of if they do not, 
whether some kind of warning is needed? 

Should there be any kind of validation of an avatar/person correspondence? 

What about “made up’ people whose faces look real but are artificially generated? 

Should they be explicitly distinguishable from real people? 

How might that be enforced? 

What standards of identity should apply? 

If chatbot helpers are created, should they have ‘verifiable pseud-identities’ so they are 
NOT confused with real people? 

Will companies using artificial assistants be compelled to identify them as such? 

Is there some measurable threshold after which an artificial bot should be treated as 
having any kind of rights? 

Would it matter whether they were corporeal or not? 

We have barely scratched the surface of a new phenomenon and its implications. 


Where will we be in ten year? twenty-five? fifty? 
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Fabien Benetou & Vint Cerf at The Future of Text 2022. Hegland, 2022. 
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Dedication 


This book, as with all my work, is intended to inspire a better future, a future where we invest 
in improving how we think and communicate. If you will indulge me, I will therefore close 
with a picture of my wonderful boy Edgar Kazu Ballard Hegland, who is six at the time of 
writing, and to whom I dedicate this work. When this picture was taken we were in Sicily and 
he had just come out of a swim in the ocean. I hope his future information environments will 


be as liquid as the ocean and as fun to swim around in. 


Frode Hegland, London late 2023 


280 


281 


e Symposium ¢ 


The Future of Text Symposium ‘23 


The Future of Text Symposium ’23 was held at The Groucho Club in London and online, 
October 2023, with presentations by David Millard & Mark Anderson, Nayanatara Prakash, 
Livia Polanyi, Vint Cerf, Gabo Beaumont with Eric Vicenti & Horacio Herrera, Leon Van 
Kammen, Fabien Benetou, Sam Brooker, Keiichi Matsuda, students from the Goethe- 
Universitat Frankfurt under Prof. Dr. Alexander Mehler (Viki Library: Shaduan Babbili, 
Kevin Bonisch, Yannick Heinrich, Philipp Stephan & Giuseppe Abrami), (News in Time and 
Space: Julian Gagel, Timo Liittig, Theresa Berg, Jasper Hustedt & Giuseppe Abrami), 


Bronnac Ferran and Frode Hegland. 
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Symposium ‘23 Participant Introductions 


Bronac Ferran is a writer and curator specialising in postwar text-based practices. She 
completed her doctoral thesis at Birkbeck earlier this year focussing on how digital and non- 
linear tendencies first found form in print, typographic experimentation and 1960s poem- 
machines. bronacf@gmail.com, more at: https://www.cloudmemory.com> 
Christopher Gutteridge — Hi, I’m Chris. I was introduced to Frode at Southampton due to a 
long term interest in how to use computers to make things better. I’ve got a history in open 
access research publishing and open data. I work as research IT support at the University of 
Southampton. In my spare time I build tools to turn real world maps into Minecraft. I’m very 
tall and currently have green/blue fingernails. (sorry, now attending remotely due to rail 
strikes) 
Jeremy Taylor (@refset) — Hi, new to the community but have been following for a long 
time. These days I work on database tech (see xtdb.com) having descended through the 
software stack after originally seeking better tools for building collaborative knowledge 
management systems (using Clojure etc.). My ‘vision’ is centered on the importance of 
planning, goal-orientation and communicating in the context of future timelines. 
Livia Polanyi (livia.polanyi@gmail.com) I am a theoretical linguist who has been working 
for(ever) on the structure of language above the sentence. My earliest work which led to the 
book 
Telling the American Story outlined the linguistic, interactional and cultural constraints on 
conversational storytelling. Later work, known as The Linguistic Discourse Model focussed 
on the structure of discourse itself .As a researcher and poet, text has never been far from my 
mind. It is a pleasure to be a part of the Future of Text community. 
Leon van Kammen I’m an audiovisual prototyper and researcher from Europe, currently 
based in Budapest. 
He is currently working on XR Fragments, in which he strives to bridge his diverse 
experiences from the worlds of audiovisual productions, programming, and the web, 
currently working on a spatial 4D URL spec at xrfragment.org 
David De Roure I’m Academic Director of Digital Scholarship at the University of Oxford, 
with a very interdisciplinary background ranging from hypertext, internet, Web Science to 
digital social research and digital humanities. Since the 1980s I’ve been a researcher and 
practitioner in music, hypertext and AI. Here is a3 minute video I made for FoT 2020 -—a 
walking tour of Oxford generated from text fragments and narrated by Alexa. Links: Oxford 
RNCM Turing 
PS I used to work on “Open Hypermedia” before the Web (see Microcosm). 
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Jonathan Finn OBE Hi — I’m a software developer/inventor of several AI creative 
applications: Sibelius (music notation/composing), strategic games (e.g. a poker AI with a 
semantic interface), and recently a high-level text editor, MindsEye. The latter has expanded 
into a ‘semantic UI’ for a mobile device or computer, using a text/graphics mix suited to the 
age of AI. Very much a work in progress... 

Mark Anderson. Hypertext researcher, primary interests: Tools for Thought, sustaining 
knowledge in hypertext/media, metadata, hypertext. Visiting Fellow at Southampton 
University post PhD in Web Science (under Dave Millard (qv) and Les Carr). Active in 
FoText since 2016, and Tinderbox community since 2004, been working primarily online 
since the late 90s. Contributory experience; 25+ working with data/metadata as a 
‘information emergency plumber’ connecting unconnected things. Also involved in ongoing 
efforts to save early hypertext systems to allow them to still be seen working today. Most 
easily found via email (m.w.r.anderson@soton.ac.uk or mwra@mac.com). 

The presentation Dave and I gave was re our Hypertext 2023 paper “Seven Hypertexts”. 
Currently free access at DL.ACM (proceedings, PDF). All 205 references from the paper as 
BibTeX (bib file via Dropbox), to aid reference easy re-use.. 

David Millard. (davidmillard.org) Prof of Computer Science at the University of 
Southampton, UK. Working with hypertext technology for 25 years, originally in the area of 
Open Hypermedia Systems and Adaptive Hypertext, and more recently research digital 
storytelling — in particular interactive digital narratives in the context of mixed reality 
systems. >> Today: Seven Hypertexts 

Amy Spencer (a.spencer@bathspa.ac.uk) Postdoctoral researcher at Bath Spa University. 
Currently working as part of MyWorld, which explores creative tech in the South West of the 
UK. As part of this, I am interested in how writers use technology in their practice and the 
future of AI in the publishing industry. 

Nayana Prakash (nayana.prakash@oii.ox.ac.uk). DPhil/PhD researcher at the Oxford 
Internet Institute/University of Oxford, studying digital storytelling in India across various 
forms. Interested in: orality/oral storytelling, “low-tech” digital storytelling, communities of 
practice. (Presentation on orality and the Internet) 

Kevin Boénisch (k.boenisch@outlook.com). Born January 1998, currently studying Computer 
Science at the Goethe University, Germany (Master’s Degree). I’ve been working as a NET 
Fullstack Software Developer for the past 4,5 years to finance my studying. Lately, ve been 
into Natural Language Processing and VR — training large language models and handling 
large amounts of data and I’m always working on side projects, which you can find on my 
GitHub. 


Keith Martin. I’ve been exploring hypermedia tools and concepts since HyperCard was 
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bundled with every Mac. My background is in print production, software development, 
graphic design, writing, immersive media creation, teaching, and debating these things with 
Frode. I work across the field of media creation and production. I was MacUser magazine’s 
Technical Editor for many years, and I recently designed and wrote content for an immersive 
media production reference web site for Meta. These days I split my focus between creating 
immersive 360 photography & video and teaching interaction design students. More is at 
https://thatkeith.com and https://mister360.co.uk. 

Keiichi Matsuda( k@liquid.city / @ketichiban on twitter) 

Designer and filmmaker / founder Liquid City. http://km.cx https://liquid.city 

Presenting some films! Have a look here. 

Latest film ‘Agents’ — https://vimeo.com/864503702 

Fabien Benetou, https://mastodon.pirateparty.be/@utopiah on Mastodon, @utopiah on X, 
fabien@benetou.fr https://fabien.benetou.fr WebXR prototypist at the European Parliament 
innovation lab. Presenting on The “Edit” button to edit not just text as content but the editor 
itself, the OS it’s running, the computer it’s running on ...and the table it’s sitting on, etc, etc! 
It’s editing all the way down. 

Theresa Berg (theresa@wenzler.org). I’m studying Computer Science at Goethe University 
Frankfurt am Main. I’m interested in Critical Computer Science and developing open source 
tools to consume news in an alternative way. Lately, I have been working on NiTS a tool that 
geolocate events on a 3D globe in VR. 

Sam Brooker s.brooker@lcc.arts.ac.uk Hey folks, I’m a Senior Lecturer at London College 
of Communications. I specialise in digital communication and in my research explore the 
relationship between digital technologies and theories of literature and culture. My 
presentation is here: https://tinyurl.com/yy6f8cmb 

Matthias Miiller-Prove (a smile and a bunch of social profiles here) Hypertext pioneer 
(Adobe GoLive Cyberstudio) and software archeologiest — open source software design 
(OpenOffice.org) — interaction designer — speaker and lecturer (social media trends, branded 
interactions) — time traveler on old maps and books (Chrono Research Lab > Chronoscope 
World) 

Brandel Zachernuk brandel@zachernuk.com I work on the proposal and development of 
spatial standards for the Worldwide web (webXR, 3D models etc) and develop prototype 
applications of such potential standards. Philosophically, I am fascinated by technological 
and sociological origins of writing and its impacts, the implications of Embodied cognition 
and the Extended Mind Thesis (EMT), and try to drag them all into any discussion of how 
this stuff works! All my identities are at https://linktr.ee/zachernuk 
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Danylo Sokol danylo.sokol2504@gmail.com | www.danylosokol.dev I’m new to this 
community. I graduated from the Technical University of Kosice in Computer Science and I 
am currently working as a frontend developer. My bachelor thesis was related to text 
(automatic summarisation of work progress reports), so I am interested in how XR and AI 
can change the way we work in the future. 

Rob Swigart Words that matter | wikipedia on Rob 

Jeremy Helm y.at/map.thinking.ear.heart Published in The Future of Text: A 2020 Vision, 
Structured Listening: Technological design to help humanize humanity http://bit.ly/ 
FoTJeremy 

Author WordPress Theme by Compete Themes 
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Al Transcript Summaries 


What follows are transcripts from the 2023 Symposium on the Future of Text, created with 
sonix.ai and cleaned up with the prompt “Please clean up this transcript:” and also “Please 
clean up this transcript of a presentation and dialog:”” when the Q&A was not taken into 
account by the AI transcript. using https://chat.openai.com where Chat GPT replied with one 
of them that “These responses have been paraphrased for brevity and clarity.” This is an 
experiment and there was a great variety in how the resulting cleaned up transcripts came 


back. Light editing has been done where mistakes were obvious. 


Frode Hegland 
https://youtu.be/U9RWcuKkpZA 
[Transcript in Full] 


I'd like to discuss our community and its tone. It's a bit odd sitting down, but let's dive into it. 
Yesterday, my six-year-old son read the poem "Now We Are Six" by A.A. Milne in school. 
Intrigued, we researched Milne and discovered he shared my middle name. Interestingly, his 
teacher was H.G. Wells. Want to know who Milne played cricket with? 

Surprise, it's a cool list, and it made me think of our diverse community. I thought of 
the Lunar Society of Birmingham in the late 1700s, and its notable members, including 
Charles's grandfather. Reflecting on our community, we aim to be social, nice, and connect 
with diverse people. Consider the lively discussions at Doug's lab, Xerox Parc, or Brandel 
Five — where the Macintosh was developed. By the way, how many know about Bentley 
Five? 

That's where the Macintosh was developed, and it's a bit embarrassing in the best 
possible way. In essence, we aim for a distributed, casual, friendly, and passionate community 
focused on the future of thought, especially with text. Today, we're exploring XR and VR, but 
not exclusively. XR helps us dream again, crucial for advancing beyond the limitations of 
current technologies. 

Let's not forget the power of traditional displays in our exploration. While VR focuses 
on immersive environments, we're agnostic about how that immersion occurs. The reason for 
the current XR emphasis is to leverage the hype, creating an avenue for passionate 
discussions — a stark contrast to the limited enthusiasm for text. Thanks for being here; your 
presence is appreciated. 


Now, moving on to XR and text's future. Text already extends our reality, and as we 
embrace XR, it allows us to dream again, reminiscent of the visionary work done in the 60s 


and 70s. Today, XR is more than just theory; people are actively using it. Our goal is to pose 
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better questions collaboratively. 

Doug Engelbart's vision of augmenting human intellect collectively is reflected in our 
diverse community. We need varied perspectives and welcome those passionate about text 
and thinking, even if we disagree. It's a call to think about thoughts in multiple dimensions of 
freedom. 

As XR becomes less theoretical, we must address questions about the document's 
nature in VR, the role of symbolic manipulation at scale, the significance of links and 
connections in an entire space, and the tools we use, considering portability and physicality in 
hypertext. 

In this journey, we need to acknowledge the realities of the day, as Danilo pointed out. 
Real-world problems, like those in Ukraine, coexist with our exploration of text and VR. XR 
is a powerful tool, as demonstrated by Yulia's piece, allowing us to manipulate and interact 
with knowledge in unique ways. 

We must be cautious about what we build in terms of tools, infrastructures, and manage 
people's expectations. Our goal is to create better questions collectively and explore the 
potential of fully interactive environments. 

In his closing, Frode emphasizes the importance of interaction, stating that without it, 
there is nothing else. Our aim is to push the fundamental aspect of interaction as far as we 


can. 


Dave Millard & Mark Anderson 
https://youtu. be/Izj CVaoPBIA 
[Transcript in Full] 
Hello, everyone. I am Dave Millard, a professor of computer science at Southampton. I am 
one of Frode’s supervisors at the [Southampton] University, and interestingly, Dave De Roure 
over in the corner was one of my internal examiners for my PhD. So that makes him a 
grandfatherly figure for me—scholarly connections abound. 

My apologies, to the few of you saw a longer version of this presentation at Hypertext 
2024, but I think it is relevant here. The paper® that Mark and I wrote to aimed to address the 
question of what hypertext is, and what is it in today’s world. Our motivation was that a lot of 
people think about hypertext only as a historical technology: something from the 1980s and 
1990s and from which we have now moved on to other things. 

At the ACM Hypertext conference, the question of ‘What is hypertext?’ and ‘What is its 
relevance?’ has cropped up again and again. In 1999, Mark Bernstein asked ‘Where are the 


hypertexts?’®+ In 2003, Peter Nurnberg gave a keynote called ‘What is Hypertext?’® and in 
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2004, Noah Wardrip-Fruin presented a paper ‘What hypertext is’, responding to Pete’s 
ideas. The general consensus is that hypertext is something related to links and linking. 
Mark and I took a different approach, and looked back through the literature to try and 
identify traditions of talking about hypertext and understanding the commonalities and 
themes, and how these might link together. We came up with seven different contexts, but I 
want to start with the punchline which I think is relevant to this community via a story about 
Jean-Paul Sartre and SharePoint. 

Jean-Paul Sartre, a French playwright and existentialist philosopher, is identified with 
existentialism. This explores the idea that existence precedes essence: we are not born into a 
world where we have a purpose, and a meaning, and a set of rules to follow. Rather, we must 
build our purpose and meaning in the world. Very often this is associated with a more 
atheistic position, but that association is not really relevant. Pertinently, if you have an 
ideology or a set of rules you are following, the point of existentialism is that you should 
critically engages with those rules, and think about what those mean in the context of your 
modern life. Therefore, people like Jean-Paul Sartre were fascinated by the concept of what it 
is to lead a ‘good life’; if there is no predefined purpose, what does that mean? 

Their answer was exactly this process of critical engagement, which they termed “being 
authentic’: if you were engaged in this process of constructing meaning and thinking 
critically, your were leading an authentic life. Conversely, to drift through life without 
engaging in that process was to live in “bad faith’—which brings me back to SharePoint. 

My university [Southampton] uses SharePoint for its information infrastructure. I am 
not a big fan of SharePoint as I do not like the way it makes me structure files, or that I 
cannot easily tag things or link between them. I cannot—despite the name—share things in 
the particular way J wish to do so. Thus for me, in my digital life, SharePoint is an example 
of ‘bad faith’. So the question for me is what technology enables me to live an authentic 
digital life? The clue is probably in the title—the answer is that hypertext lets me live an 
authentic digital life. 

Hypertext is a mechanism that enables us to engage with information, build the 
structures and rules, and think the way we want to think: it is an empowering technology. Ted 
Nelson, nearly 50 years ago, said “You can and must understand computers NOW”®’. This is 
because he was very clear that this would define the way we communicated and thought. 
Taking control and ownership of it [computers] was really of the utmost importance. 

How does hypertext enable us to do this? It is to do with linking, but it is also what you 
do with those links and the fact you can then use those links to construct other structures so 
you can engage in the way you want to engage. SharePoint gives me hierarchies, which is 


fine, but there are other things I might want to do and hypertextual technologies enable us to 
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engage with information in the way that we want. 

This comes down to a principle of non-regular structure, enabling us to be inventive 
with the structures we use, taking me a a second French philosopher, Derrida, who is 
associated with post-structuralism. Structuralism is the idea that we understand the world 
through grand structures and meanings. The post-structural movement does not abandon 
structure completely. It contends there are different structures—different ways of looking at 
the world—with which you can engage. Derrida said that if we enable this [thinking], take a 
position, and think about the world in these different ways it allows us to engage in what he 
calls ‘jeu libre’ or ‘free play’. 

I think essentially what we are saying about hypertext is that it is a set of technologies that 
use non-regular structure to build meaning in our informational lives and engage in free play 
in our digital information systems. 

When Mark and I looked at the literature and saw these traditions of hypertext, we saw 
this philosophy being applied to different areas of life: to knowledge, to software, to learning, 
to interactions, creativity, sociality and play, where it manifests in different ways. I think what 
we have in Extended Reality is a new space in which we can take that hypertextual 
philosophy and try to apply it. Some call this ‘hypertext as method’®®, that hypertext is a way 
of thinking about the world that you can apply in different spaces. 

I will not run through the detail of these various areas [of hypertext] in detail—they can 
be read in our paper®?. Essentially, there is long history of papers that engage with knowledge 
and learning, systems that look at hypertext as a sort of infrastructure, as different types of 


interactions, and of how hypertexts manifests in social and creative spaces 


I am very struck by the hypertext we see in the wild—what has sometimes been termed 
‘feral hypertext’. We had this explosion [of hypertext] in the ‘web 2.0’ era, and we see 
another today. We see people talking about ‘human in the loop’ for AI. We see a new 
generation of Personal Knowledge Management tools and systems. We see AR and VR, and 
what hypertext means in those types of context—this new kind of canvas the Frode was 
talking about. 

So, we are clearly at a point where there is an opportunity to take this hypertextual 
thinking and apply it in a new space. At the same time, there are things that have been left 
behind. Hypertext is not just about links, link following, and these kinds of structures: it is 
also about more ambiguous structures like spatial hypertext and spatial hypermedia. There is 
this whole idea about how hypertext interacts with Artificial Intelligence, how it manifests in 
adaptive hypermedia, hypertext literature and games. And, the notion that hypertext doesn’t 
kill people—people kill people: we have problems in our hypertextual spaces. There we have 


social problems, such as bad actors and [the challenge of] how we defend our technologies 
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and ourselves in those spaces. 


In conclusion to the idea of ‘what is hypertext?’, in our survey we see that hypertext is 
really a mindset, rich with ferality. By this, I mean that we are not the only ones talking about 
this and that there are others out there world actively pursuing this subject: you see it 
manifesting in many different domains in many different spaces. 

Hypertext is very much alive, and has things to say, and lessons for us all to learn. The 
question for this community is that we can see some of these connections that others cannot. 
How then do we reach across the divide, help people make those connections and learn from 
our shared history? 

Thank you. 

Frode Hegland: Any questions? 

Leon Van Kammen: Do you think that hypertext is a form of hypermedia, or is hypermedia 
a form of hypertext? In other words, is there a distinction? 

Dave Millard: Generally, people in this space use the terms interchangeably. When I talk 
about hypertext, I mean hypermedia. Very early systems were textual (out of necessity), but it 
is about presenting information in a form suitable for people. It is information for people, 
going back to Doug’s idea of augmenting human intellect. 

Brandel Zachernuk: Describing hypertext as a frame of reference is exciting. Does applying 
these lenses help define what should be included or excluded in an authoring environment? 
Dave Millard: Different domains require different approaches. There are common 
challenges, and solutions may manifest differently. It is about helping people cope with the 
freedom hypertext offers and dealing with the challenges of a non-predefined way of 
working. 

Mark Anderson: Tools should not have too much opinion. Tools should leave space for 
exploration, especially in hypertext, which is inherently freewheeling but people need to be 
present in the authoring process. The rules are not set, and tools should allow for nuanced 
questions. 

Fabien Benetou: In virtual reality, there is a translation of hypertext and hypermedia, 
especially with transclusion. Does this apply in XR, going from one virtual world to another? 
Dave Millard: Yes, in my current research on mixed reality systems, we view mixed reality 
as physical hypertext—information overlaid on physical or virtual environments. XR allows 
us to push the boundaries, exploring worlds within worlds. 

Frode Hegland: Highlighting that hypertext as a tool for thoughts and presentation is not the 


same is crucial. It liberates us to create freely while thinking. Thank you, guys. 
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Nayana Prakash 
https://youtu. be/G-X1K Ox Ux28 
[Transcript in Full] 


Hi, everyone! I hope you can all hear me, both online and in person. Today's presentation will 
be different and not too long because I don't have all the answers; I have a lot of questions. 
I'm Nayana, a PhD student at the Oxford Internet Institute, focusing on the social science of 
the internet. My research is about digital storytelling, specifically orality and the internet. 
Orality, in my context, refers to spoken language. My PhD explores digital storytelling in 
India, where oral storytelling is a crucial tradition across languages, preserving history and 
community experiences. It's audience-specific, relying on the live, in-person performance to 
convey the story's essence, changing with each telling and shaped by the environment. 

The recent push in India to archive and digitize oral storytelling, like the partition 
stories from 1947, reflects a realization that firsthand witnesses might not be around much 
longer. This effort seeks authentic ways to collect and share stories, especially considering 
varying literacy levels. 

In non-archival contexts, platforms like WhatsApp's voice note function gain popularity 
in India, offering a way to communicate beyond language barriers and literacy constraints. 
My focus is on how orality on the internet reshapes intimacy, space, and closeness. 

So, how does orality reconfigure space, intimacy, and closeness? 

Orality on the Internet: 

I'm curious about whether orality is the future of internet communication. I explore its 
use as an archival tool and ponder whether true orality is rare, given our reliance on written 
words. Is orality linked to the past, or does it have a future role in communication? 

I'm particularly interested in moments of orality and sound beyond language, resisting 
uniformity in online communication. I'd love to hear your thoughts and questions. 

Dave Millard: Considering oral storytelling, what strikes me is its interactive nature, 
adapting to the audience. How does this interactiveness manifest in non-interactive 
technologies on the internet? 

Nayana Prakash: The dynamic nature of storytelling, audience-dependent and responsive, 
changes with time and platforms. The flattening of stories for a broader audience and the shift 
from live to asynchronous experiences also influence the nature of oral storytelling. 

Anne Allen: Drawing parallels with historical European recordings of folk stories and music, 
how can we utilize oral recordings for the future? The challenge is organizing and accessing 
them efficiently. 


Nayana Prakash: Archiving voice notes brings up intentional and unintentional archives, 
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raising privacy considerations. There's a need for better tools to link and organize these 


recordings effectively. 


Vint Cerf: Reflecting on the power of language, he wonders about the role of dialogue in the 
future of text, especially in exploring connections between various media types. 

Nayana Prakash: Emphasizing the need to embrace all forms of communication on the 
internet, my focus is on facilitating a broader dialogue across different cultures and 
languages. 

Brandel Zachernuk: Oral traditions historically allowed for the reinterpretation of messages 


with each retelling. How does digitization affect this transient nature of orality? 


Nayana Prakash: The digitization of orality raises questions about the permanence of 
traditionally impermanent forms. It doesn't replace other forms but widens the scope of 
communication. The tension lies in the balance between capturing something before it's gone 


and the potential domestication of living traditions. 


Vint Cerf 
https://youtu.be/Ct70ZzWxxpA 
[Transcript in Full] 


Thank you. I'll adjust my position to get closer to the microphone. This presentation is going 
to be kaleidoscopic, delivering a variety of loosely connected ideas. I hope you'll embrace 
imagination kaleidoscopically even after I'm done. 

Now, let's talk about the future of text. When considering the pasta equivalent for texts, 
my whimsical thought is alphabet soup, pun intended. Moving on, we discussed hyper X, 
where X can be various values. What about hyper sound and hyper imagery? Do these 
concepts hold meaning, and should we explore how hyper content is created? Authors might 
use hyper capabilities to guide readers, suggesting associations. On the flip side, consumers 
may desire tools allowing the exploration of unintended associations. This dual aspect is 
crucial in the creation and use of hyper X. Frode’s work on text creation tools, like the visual 


meta component of PDF, aims to enhance text utilization. 


Frode's tools, including reader and visual meta, demand skill for effective use. 
Watching Frode's demonstration resembles playing a musical instrument. While these tools 
are compelling, the challenge lies in ensuring they aren't too complex, akin to learning a 
musical instrument. We must ponder how to make these beneficial tools more user-friendly. 
Generative Al's speed raises interesting possibilities for real-time storytelling. Imagine a story 
unfolding with generative AI, where illustrations seamlessly appear. People are already 


composing stories with generative AI, and real-time narratives may not be far off. 
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Considering computer-based tools, our focus has been on organizing, assisting, and 
communicating thoughts. Text writing, in particular, offers a unique aspect—the ability to 
review, correct, and change. Similar to computer chipsets, writing with memory expands 
functionality. This time component in writing is intriguing. 

Lastly, the impact of ChatGPT on the world has led to a new tool—prompt engineering. 
This discipline involves crafting prompts for ChatGPT, with the "window" size determining 
interesting results. Prompt engineering, akin to conceptual calligraphy, allows text generation 
with the assistance of chatbots. 

I must conclude, looking forward to your discussions. My apologies for leaving early, 
tasked with summarizing two days of Royal Society discussions on science for justice. Thank 
you, Frode, for organizing and contributing to these concepts. 


Mintter 
https://voutu.be/tVnUVaey9_M 
[Transcript in Full] 


Gabo Beaumont: Thank you very much, everyone, and Frode, for having us here. I'm Gabo 
Beaumont. We are from Minter, and we are going to present what we have been working on 
for the last almost four years. As I discovered hypertext four years ago, I was struck by the 
concept, surprised it's not more common in human society. It is our medium for thinking and 
communication. That's big, and we are fascinated with it. We also discovered that the web, as 
a hypertext system, has deep technical limitations, making it open for publishing but not 
decentralized for collaboration. Collaboration requires going to central platforms due to the 
web's technical limitations. All these central platforms function like central planning and 


aren't efficient enough for the complexity of digital societies. 


What we are working on is an open hypermedia system for the web with the vision that 


intellectual communities should run the internet, not central platforms, by syncing computers. 


We have built an open hypermedia system called Minter, involving a local-first 
application. It runs offline, and you can also run a Minter site on a domain. It focuses on open 
collaboration, allowing diverse intellectual communities to collaborate without relying on 
central platforms. The system supports cryptographically signed groups, creating a 
hierarchical multimedia editor where users can add various media types, such as images, 
audio, video, files, PDFs, etc. Contacts are direct connections with peers, and content is 


synced based on the connected people. 


Every entity in a Minter document is an opportunity to create a unique URL. Users can 


create, edit, and version documents with digital signatures, ensuring authenticity and 
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immutability. Contributions from different authors are tracked, and documents are version- 
controlled, inspired by Git. Links between documents are transcludable, creating a 


decentralized and non-breakable linking structure. 


The content is stored locally first, ensuring permanence and easy indexing of links and 
references. The system also integrates micropayments using Bitcoin to distribute value to 
authors based on contributions. 

The ultimate goal is to amplify intellectual communities, improve authorship 
recognition, and fix the broken business models of academia and media, making them more 
accessible and collaborative. 

Audience Questions: 

Decentralization and GDPR: The immutability of the system raises questions 
about GDPR and the right to be forgotten. While the system doesn't impose GDPR measures, 
it emphasizes the importance of open collaboration, allowing content to be stored indefinitely 


once shared publicly. 


Web-based vs. Native Client: The Minter system is open source (available on 
GitHub), and while it currently uses a desktop application for local-first capabilities, the team 
is open to exploring ways to create a bridge between local nodes and web browsers in the 
future. 


Use Cases and Impact: The primary use case is open collaboration among 
intellectual communities, fostering sharing of knowledge among strangers. The impact is 
expected to improve authorship recognition, encourage more people to share, and fix the 
broken business models of academia and media. 

Payments and Business Model: The system currently uses Bitcoin for 
micropayments, and the team is open to adding other currencies. The business model aims to 
fix the broken systems in academia and media by improving distribution of funds and making 
editorial processes more effective. 

In conclusion, Minter aims to be a catalyst for open collaboration, decentralized authorship, 


and a more sustainable model for intellectual communities. 
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Leon Van Kammen 
https://youtu.be/XuNVU7uTovo 
[Transcript in Full] 


Thank you all for joining me today. I'm Leon, a creative technologist, and I want to delve into 
the concept of the hypermediatic experience, a term I coined to redefine hypermedia through 
the lens of XR (Extended Reality). While hypermedia seemed a bit outdated to me during my 
research, I found inspiration in the potential of XR, a 3D space that could reshape our 
understanding of text. 

I pondered the origins of text and images, and similarly, the relationship between 
hypertext and hypermedia. It felt like they should be interconnected, much like the 
relationship between URLs and hypermedia, transcending the traditional HTML paradigm. 

I questioned the conventional explanation that hypertext can link infinitely, while 
hypermedia seemingly has only one outbound link. In my view, hypermedia, which includes 
videos, images, and audio, should be more interconnected and not confined to single 
endpoints. 

Drawing parallels between reality-to-text synthesis and XR, I conceptualized a text 
synthesizer, where the URL acts as a carrier, modulated by hypermedia. The challenge was to 
make this hyperconnected, unlike the current fan-out pattern seen in hypertext. 

I proposed the idea that URLs might be the real driving force, acting as textual carriers 
transporting information between brains. Hypermedia could encompass anything, even 3D 
objects, fostering a truly interconnected experience. 

I expressed concerns about platforms like TikTok and YouTube being great video 
databases but not truly hypermediatic. Their algorithmic curation limits user control, unlike 
the early web rings. 

To achieve a hypermediatic synthesis, I suggested embracing URLs as the focal point, 
decoupling from the HTML paradigm. This led me to envision a future where apps and 
operating systems are browsers, interconnected through URLs, offering a seamless 
experience. 

The whole 3D scene has been collapsed into an XR word graph, which is text. So we 
went from complete 3D spatial to text, allowing associations to be drawn visible links. If you 
click on a cube, you can see the hash cube appears in the URL, and visible links start to 
appear with anything cube-related. If it's a button doing something with a cube, an image, or 
another 3D document embedded with a cube, all these things can be shown in the 
relationship. So thank you very much. Any questions welcome. 


Frode Hegland: First and obvious question. And then over to Dave, what happens with the 
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domain part of this? You know, domains die. So how does that function then when you 


change domain or something goes down? 


Leon Van Kammen: That depends on the publisher. This Excel fragment spec is protocol- 
agnostic. It can work with any protocol. If you're very serious and worried about this, you 
have to choose something like IPFs. If you're not so worried or want to be more EU right to 


be forgotten or just be lazy, you go for HTTP. You upload something to another protocol. 
Dave Millard: Okay. So yeah, it was really, really good. Thank you very much. This is more 


of a comment. To answer some of the questions you raised. We tend to look at hypertext 
today through the lens of the web, which became a dominant hypertext platform. If you go 
pre-web, you find interesting work on this space. The place I would go is probably open 
hypermedia systems. Not open as in open source or free, but open as they separate hypertext 
structure entirely from content. Microcosm, for example, allowed you to take arbitrary 
content like a video, apply hypermedia on top of it. Microcosm in '91 allowed you to play a 
video, follow a link in the video to an audio file or link in the audio file to a word file, etc. 
Images were all completely hyperlinked. You can still run it. It was all about combining 
different hypermedia structures dynamically. This work culminated in structural computing, 
pushing it into the operating system layer. The web crushed this work in the mid-2000s, but 
now we're on the other side, able to think about these things again. XR environments, I see as 
a modern manifestation of open hypermedia. That's what I'd love to see. Open hypermedia 
systems. 

When you take the hypermedia structure away, you can take the same content and apply 
alternative hypermedia structure on top of it. An adaptive hypermedia system allows 
personalized hypertext views independently of content. There are alternative structures like 
spatial hypertext structure, taxonomic hypertext structure. Systems can manage and overlay 
these structures simultaneously. Component-based open hypermedia systems involve 
structure servers, link servers dynamically combined. This movement culminated in structural 
computing. The web tsunami in the mid-2000s suppressed this work, but now we see some of 


these ideas resurfacing. 


Dave Millard: It gets better because you can take an alternative hypermedia structure and 
put it on top. Link structures are one type of content, but there are alternative structures like 
spatial hypertext structure, taxonomic hypertext structure. Systems can manage and overlay 
these structures simultaneously. Component-based open hypermedia systems involve 
structure servers, link servers dynamically combined. This movement culminated in structural 
computing. This work was suppressed by the web, but now we see some of these ideas 


resurfacing. 


Leon Van Kammen: That was a fantastic record. 
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Dave Du Roure: We were working on open hypermedia. We wrote papers about it at the 
first web conference, but it wasn't included because people wanted to hear about the web. 
Why didn't it catch on? If it's better, why didn't it catch on? We tried to tell the web 
community how to do this, but the web was just crushing this work. It required more 
engineering. The simplicity of the web and people not expecting link integrity made it catch 


on. Now, with current technologies, these engineering challenges are more accessible. 


Dave Millard: As soon as you store link data structures separate from documents, you need 
more sophistication. The web came along with a simple solution, and it scaled brilliantly. We 
went, "That's terrible, you threw out all the great stuff!" But the web said, "We don't care, we 
have 100 million users." Your system is the first I've seen as a genuine alternative, scalable 
infrastructure to the web that can deliver what was available in more centralized desktop 
systems of the '90s. 

Dave Millard: My concern is how to deal with the commercial and social reality of the web, 


walled gardens, and apps. XR environments give us a fresh canvas to rethink things. 


Frode Hegland: | have comments and questions. Doug Engelbart's system was very 
hierarchical. The idea of using links to put commands into nodes was explored. Using the 
link structure to specify views in the URL was an interesting idea. I want to highlight the 
super importance of addressability and things being able to say what they are. Visual meta is 
a simple example. When you collapse addressability, it brings back voice and gives an 
opportunity to rethink things. Questions and comments? 

Eric Vicenti: This is really tied up with what Vint said before, that when they were creating 
the internet, there was no commercial incentive. Now we are in an era with no commercial 
constraints. XR is a free space, an opportunity to spark more ideas and better solutions. This 


project decision allows more choice and freedom. 


Leon Van Kammen: | want developers, game developers, web developers to realize that 
every entity in a 3D document is an opportunity to create a unique URL. Everything inside a 
3D document has a URL already, making it addressable by default. You can reference and 
include 3D documents efficiently. 

Frode Hegland: XR environments are the perfect open hypermedia environment. It's 
connections and associations held separately, dynamically combined at runtime to create a 
personalized, interlinked experience in XR space. 

Dave Du Roure: Why didn't these ideas catch on? We were working on open hypermedia. 
We wrote papers at the first web conference, but it wasn't included because people wanted to 
hear about the web. The simplicity of the web, combined with the lack of expectation for link 
integrity, made it catch on. It required more engineering, and the web's simplicity won. Now, 


with current technologies, the engineering challenges are more accessible. 
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Dave Millard: People didn't realize it's a killer app thing. The web did one thing brilliantly: 
scale. None of these systems scaled well. The simplicity of the web, the fact that people didn't 
care about link integrity, made it catch on. Now, with current technologies, these engineering 


challenges are more accessible. 


Vint Cerf: I think there's a very interesting issue here. You're demonstrating a very dense 
hyperlinking structure. The 3D environment provides an opportunity for simultaneous 
parallelism. One way to see this is that, in the 2D world, we got good at linking text and 
making it clickable. You're doing the same thing in a 3D environment. It's an exciting 
opportunity. The really hard part is making it scale. 

Frode Hegland: If you could, in a sentence, tell people what they should remember from 


your presentation, what would it be? 


Leon Van Kammen: Every entity in a 3D document is an opportunity to create a unique 
URL. 


Livia Polanyi 
https://voutu.be/X5SHOIOIg8mE 
[Transcript in Full] 


Hello, I'm Livia Polanyi. Many years ago, I wrote a book called "Telling the American 
Story," delving into the constraints on oral and conversational storytelling—linguistic, social, 
interactional, and cultural. Exploring how this works with actual storytellers versus everyday 


conversational storytelling is intriguing. Orality becomes a focal point in this discussion. 


In this presentation, the term "letters" encompasses both written characters or symbols 
and actual letters people send. Reflecting on the historical progression, we moved from 
cuneiform technology in ancient times, primarily accessible to a few, to employing scribes for 
letter writing in subsequent eras. The scribe's role persisted even into contemporary times, 
especially for illiterate individuals. However, literacy spread, leading to advancements such 
as lightweight pens, paper, typewriters, and dictaphones by the late 19th to early 20th century. 

Fast forward to today, where we've transitioned away from many traditional letter- 
writing technologies. People no longer buy stamps; instead, letters are printed or sent via 
email. Considering this, I pondered when I last received or wrote a substantial letter, distinct 
from formal announcements or legal documents. Internationally, much of our communication 
has shifted to text, particularly among children growing up in digital environments. 

Looking ahead, as virtual reality (VR) gains prominence, I'm fascinated by the 
multitude of proposals for VR keyboards. However, the concept of composing text one letter 


at a time in this space seems counterintuitive. In the realm of increasing conversational AI 
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and oral interfaces, relying on archaic keyboard input in VR struck me as odd. 

I come from a linguistic background and have spent considerable time in the NLP 
community. Despite being touted as a computational linguist (a humorous misconception), I 
lean towards orality in this context. Speech input, facilitated by speech recognition and 
natural language processing, holds promise. While keyboards might still offer value for 
editing, the idea of entering text one letter at a time in VR seems impractical. 

Nayana Prakash: Thank you for your insights, Livia. Your perspective on the evolution of 
communication technologies, from orality to contemporary digital environments, raises 
essential considerations for the future, particularly in the realm of VR. As we navigate these 
changes, understanding how they impact dialogue and conversation becomes increasingly 
crucial. 

Fabien Benetou: I'd like to share my experiences working on XR prototypes for knowledge 
management at the European Parliament. While the current keyboard-centric approach to VR 
text input works, I find it limiting and suggest exploring alternatives such as micro gestures 
or speech. However, avoiding fallback options like physical keyboards is vital to encourage 
innovation and prevent reverting to familiar but outdated methods. 

Livia Polanyi: Innovating new affordances is essential when transitioning to new 
technologies, instead of imposing old solutions. This adaptation becomes crucial for 
capturing the potential benefits of emerging tools. 

Frode Hegland: I appreciate the insights shared today, emphasizing the need to shed 
outdated terms and consider the most effective ways to leverage evolving technologies. 
Gratitude to the speakers for contributing to this dialogue on the future of text. 

Brandel Zachernuk: Livia, your mention of the scribe's role as a communicator and 
message-crafter is intriguing, especially in the context of future technologies. Drawing on 
contemporary scribal traditions involving diagrams and illustrations, we can reimagine these 
intermediary functions in a technologically advanced society where everyone has similar 
capabilities. 

Livia Polanyi: Absolutely, Brandel. It's essential to acknowledge the potential risks 
associated with technologies like ChatGPT, which might be perceived as digital scribes. 
While they offer vast knowledge, the human touch in crafting messages should not be 
overlooked. The scribe's historical role in ensuring accurate transcription and aiding 


communication remains valuable. 


The dialogue concludes with gratitude for the insightful discussions and a focus on 


reimagining intermediary functions in a technologically advanced society. 


301 


Fabien Benetou 
https://youtu.be/TPiSnP1L6Vs 
[Transcript in Full] 
I'm Fabien Benetou, and I do Web XR prototyping, mainly for virtual and augmented reality 
on the web. I work with the European Parliament for knowledge management. Writing down 


ideas is how I manage, but using a small screen is not effective. I started exploring other 


methods. These are my slides; I'll need to share. Just click. 

So, last year, I made a short presentation for the future of text on interfaces. Everything 
around us is an interface. Affordances toward information digitally or the handle of a door. 
My conclusion was everything is an interface. The question then is, is it useful to say that text 
is an interface? Having a discussion is an interface to a concept. My life is being a prototype. 
I want to build stuff, be pragmatic. Do I need to build an interface? 

Keeichi Matsuda: So, do you have any other examples of things that edit themselves? 
Fabien Benetou: I'm working on a VR self-editing system, a system where everything is 
editable. Responsive programming, changing the device through the device itself. Digital and 
non-digital, inspecting different layers with a headset, editing the edit button itself, having an 
edit button on yourself. It can be done socially, sharing metadata online on how to edit things. 
Frode Hegland: Interesting idea. Do you have examples of graffiti? 

Fabien Benetou: I have my own wiki, a network of wikis. Graffiti is a form of editing, 
challenging structures in places. 

Keeichi Matsuda: How far have you run this thought experiment, especially regarding 
permissions and political issues of editing? 

Fabien Benetou: It's a revolutionary idea, challenging power structures. It's not just about 
editing objects but understanding layers, physics, chemistry, social aspects. 

Frode Hegland: Creativity and involvement are crucial. Mathematics itself is editable. 
Keeichi Matsuda: What about the right to repair? Is there a connection to your thinking? 
Fabien Benetou: The right to repair is a first step toward not accepting finite designs. It's 
going in the right direction, allowing people to fix their devices and have the tools to repair. 
Leon Van Kammmen: How does your thinking connect to the right to repair? 

Fabien Benetou: The right to repair is like a first step toward more, not accepting that the 


design is finite. In France, for example, you have to sell the device with the tools to repair it. 
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Keeichi Matsuda 
https://youtu.be/vDSVqgLeqk0 
[Transcript in Full] 
Hi, everybody. Thank you, Frode. Frode and I go way back. We met yesterday, I think. Not 


really. Yesterday he sent me a message on Twitter saying, do you want to come to this thing? 
I said, okay, so here I am. Very nice to meet you all. I'm a designer and filmmaker based in 
London, East London. I'm going to show you some projects that resonate with the event's 
themes. My background is in architecture, and I got interested in the possibilities of creating 
composite environments blending physical and virtual. Augmented reality caught my 
attention, bridging physical spaces and virtual possibilities. I have two themes to discuss: 
Hyper X and Agents. 

In 2009, I started exploring the idea of media escaping 2D surfaces, forming navigable, 
interactive, and mutable information environments. I envisioned layering virtual onto 
physical spaces, even replacing physical environments. I questioned who creates this media 
and advocated for users leaving their traces. In 2016, I continued exploring these ideas in 
"Hyper Reality," imagining a city with virtual layers for each shop. These films serve as 
provocations, urging us to think about the future we're creating. 

In 2017, I delved into virtual reality and artificial intelligence in a film exploring 
various information environments for work. Last year, I built a real installation for Niantic, 
overlaying a conference venue with virtual layers for navigation, agenda display, and more. I 
focused on utility and avoided extractive business models. 

The second strand is Agents. I started thinking about new interaction modalities, 
exploring how our bodies could be interfaces. However, I shifted towards conversational UI, 
imagining AI agents coexisting in shared worlds. My recent film, "Agents," explores diverse 
Al agents with different functionalities, capabilities, and alignments. I introduced the concept 


of an animist model for AI interaction, moving away from superintelligent, omnipresent AI. 


We developed a prototype called "Wall," a domain-specific AI with character and 
expertise. It engages users in a learning experience, emphasizing curiosity. These projects 
serve as provocations to make us question the future we're building. I'm not prescribing a 
utopian or dystopian vision but encouraging critical engagement with potential futures. 
Dave Millard: Thank you for the inspiring work. I've used "Hyper Reality" as an example 
for years. I see it as a warning about the ethics of corporatizing space. What's your intention 


with your work, and what reactions have you received regarding its ethical implications? 


Keeichi Matsuda: | aim for provocation, not pushing a specific vision. I present what seems 


likely based on current trends, encouraging people to question and envision alternatives. My 
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intention is not to force a perspective but to engage people and spark discussions about the 
future. The impact has been positive, opening doors to interesting conversations and allowing 


others to explore these ideas. 


Frode Hegland: What is space in the context of subjective space, considering different 
people may experience different environments through various devices? 

Keeichi Matsuda: I refer to it as subjective space, acknowledging that different filters and 
devices can create varied experiences of the same environment. The challenge is addressing 
the political aspect of shared spaces, deciding who has permission to edit, enter, or see the 
space. This requires engaging with political issues head-on and envisioning how shared 
computing systems should function. 

Brandel Zachernuk: Your shift from instrumental use of technology to agentive is 
intriguing. Is it to avoid the complexity of instrumental interfaces or due to a preference for 
more agentive interactions? What capabilities have you observed with agentive interfaces? 
Keeichi Matsuda: The shift stems from a desire for simpler, more human interactions, akin 
to historical advancements in computing interfaces. While instrumental models have practical 
uses, agentive interactions align with natural language processing and a conversational UI. I 
seek to simplify interactions and introduce more human-centric models for engaging with 
technology. 

Leon Van Kammen: Your latest film evoked an emotional response, especially regarding the 
contrast between one powerful agent and many diverse agents. Can you share more about the 
emotional journey you intended for the audience? 

Keeichi Matsuda: The emotional journey revolves around the core theme of one versus 
many. It explores the choice between a single powerful agent aligned with corporate values 
and a community-driven model where users become both producers and consumers. 
Characters represent the blind faith in a singular powerful agent versus a community-driven, 


pantheist approach. The film aims to provoke reflection on the one-versus-many dilemma. 
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Sam Brooker 
https://youtu.be/8GdSabCCLk& 
[Transcript in Full] 


This essay discusses modern railways and rituals at two different points, with various topics 


in between. 


Let's start with some fundamental definitions. 


We may find it useful to remind ourselves that medium just translates as a means of 
communication, encompassing such forms like Paul Revere's midnight run to warn the 
American colonial militia about imminent arrival of British forces in 1775, North Korean TV, 


Gondor's signal fires from Lord of the Rings, or Pravda as a news source. 


All these examples serve to collapse distance between source and destination, 
transmitting information beyond the range possible by human agents without augmentation. 
This approach aligns with James Carey's transmission model of communication, where 
messages are distributed to control distance and connect people. Through this text, for 
example, I am able to speak with you across space and time, providing information (recall too 
that information means to give form; shaping your thinking, behavior, attitudes, and culture). 
We may also think of the global spread of Christianity in this regard, or colonial influences 
still felt today. 

This transmission model is just one way to conceptualize communication, however. 
Carey discusses an alternative view, but before diving into this let's touch on how we are 


defining technology. 


Technology we can describe as the application of scientific knowledge to the creation 
of tools, systems, and solutions. I won't provide examples here, since it seems self- 
explanatory, but will instead use this space to discuss the impact of technology on social 
processes, behaviors, and culture. As we have known since McLuhan, ‘personal and social 
consequences of any medium - that is, of any extension of ourselves - result from the new 
scale that is introduced into our affairs by each extension of ourselves, or by any new 
technology.’ 

The invention of a technology and its subsequent cultural and social significance 
inevitably has a time gap; for instance, TV and film, where the former really impacted culture 
some thirty years after its initial development (though film had a short lag time). Equally the 
impressiveness of the technology does not determine its impact. Technological innovations — 
and their consequences for society — more often configure using off-the-shelf components in 


innovative ways, as Clay Christiansen suggests. Consider Netflix or — as we will here — VR. 
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In each case the potential for these technologies is realised through the configuration of 
existing elements. 

A technological determinist might argue that functions follow after forms, as smoke 
signals and the written word can convey different complexities of information (an idea 
Postman offers). Edison's quote about making electricity cheap, rendering candles a luxury, 
emphasizes the role both of cheapness and accessibility in the success of a technology, but 
also the way it alters our relationship with precursor technologies which ostensibly serve the 


same function. 


In this respect we may consider Apple’s VR a potential game-changer. It is cheaper, 
offers exciting innovations, and does the job of previous technologies better and with less 
friction. However, the challenge lies in understanding what exactly VR disrupts and whether 


it's a magnificent platform in search of a purpose. 


This brings us to Carey’s other form of communication: the ritual. Rituals are, as Carey 
describes it, ceremonies that unite people in fellowship and commonality. Unlike the 
transmission model, rituals don't necessarily impart information but instead create a sense of 
community. For instance, a royal wedding brings people together without providing 
informative content, simply around a shared idea of culture and nationhood. 

Carey’s distinction between transmission and ritual communication prompts questions 
about VR. Can it go beyond merely closing gaps and become a platform for meaningful 
rituals? It excels in transmitting information but struggles to promote its ritualistic side — see, 
for example, the rather dispiriting images of solitary people sitting in dimly lit rooms, 
navigating an information space nobody in their physical environment can see. 

The uniqueness of media businesses lies in their wide-ranging role as means of 
accessing culture, communication, and facilitating rituals. Technologies, like model railways 
or Minecraft, are often beloved by enthusiasts despite their flaws, enthusiasts who struggle to 


convey their excitement to less motivated parties. 


The challenge, then, is moving people from the "wow" of VR to the "why," 
understanding its specific purpose and value. How do we bridge the gap between novelty and 
integration into daily life? This will remain an ongoing question as VR evolves, and one that 
will help shape the future of text. 

Frode Hegland: Before we move to questions, a quick note on technology and its role in 
mixed reality environments. The shift from the medium to the things within the medium is 
becoming interesting, as exemplified by the Pokemon convention where people participated 
in a ritualistic activity within a local park. This demonstrates the potential of technology 


facilitating shared connections in unique ways. 


Dave Millard: Your presentation is fascinating, and I appreciate your insights into VR's 
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potential. I wonder if VR could be a solution to specific problems, especially considering its 
timing in history. For instance, the challenges with retaining digitally native documents could 
find a solution in VR, offering a different way to handle vast quantities of data. 

Fabien Benetou: I recently conducted a VR workshop with kids aged 11 to 12. The entire 
ritual was about playing, embracing a mindset of exploration and excitement. This mindset of 
"let's play" might be a universal approach to VR, highlighting its potential to engage people 
without overthinking its purpose. 

Brandel Zachernuk: Your concept of VR as a solution looking for a problem is not unique 
to this technology; many innovations find their true purpose after the fact. As seen in the 
evolution of Apple products, the initial positioning is often an accurate reflection of what 
people believe the technology will be. VR's potential and purpose may become clearer as 
time progresses. 

Mark Anderson: I'd like to emphasize that VR could potentially address problems we're 
facing, such as data overload. The timing of VR's emergence might align with our need for 
innovative solutions, even if we're not entirely sure of its specific purpose yet. 

Fabien Benetou: Reflecting on a recent VR workshop with kids, the ritual was simply about 
playing. This mindset of "let's play" might be a universal approach to VR, highlighting its 
potential to engage people without overthinking its purpose. Frode Hegland: We have more 
questions coming in, but due to time constraints, we'll move on to the next presentation. 
Thank you, Sam, for your insightful discussion on the complexities of communication and 


technology, particularly in the context of VR. 
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Frode Hegland 
https://youtu.be/gL-UYuRIc_Y 
[Transcript in Full] 


Today, I'll discuss two main topics: Visual-Meta and Author and Reader. Regarding metadata, 
let me clarify my definition. Metadata is information that defines what data is. For example, 
every document has a title, and metadata indicates that it is the title. In this presentation, I'll 
focus solely on document metadata, excluding topics like government surveillance. Now, let 
me quickly revisit document metadata. It defines what the document is, including its inherent 
metadata like the document's heading, and appended metadata, such as usage instructions and 
legal information. I'm not delving into the more complex aspects like expensive secondary 
metadata for this discussion. 

Now, moving on to Visual-Meta, which essentially is a way of representing document 
metadata visually. It includes the type, font size, and additional details. This visual 
representation is integrated into the manuscript and is a core aspect of my entire PhD, 
condensed into the BibTeX format for readability by both computers and humans. 

The next part involves a demonstration of two software tools: Author and Reader, both 
designed for macOS. While they are paid software, I offer free access to Mac users upon 
request. I'll showcase features like copious citation, powerful views, library function, 
interactive map, and glossary. Copious citation is a simple process of copying a citation from 
one document and pasting it into another using Author and Reader. This facilitates seamless 
referencing and navigation between cited documents. Additionally, I emphasize the exciting 
development that allows BibTeX compatibility, enabling collaboration across different 
platforms and systems. I showcase powerful views by displaying a textbook organized by 
color-coded headings, offering a visual aid for easy navigation. The library function allows 
efficient document navigation and the ability to jump to specific sections quickly. 

The interactive map and glossary features are illustrated using the concept of defining 
connections and relationships between different concepts within a document. This visual 
representation is further utilized to generate a glossary, providing an enhanced understanding 
of the content. 

Addressing the potential use of Visual-Meta in extended reality (XR), I present a 
mockup of Reader in a virtual reality (VR) environment. The goal is to explore new ways of 
reading and interacting with information in a spatial context, considering the evolving 
capabilities of XR. 

In conclusion, Visual-Meta aims to address the limitations of traditional metadata 


representation by incorporating a visual element. The tools Author and Reader offer practical 
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solutions for managing document metadata, citing references, and enhancing the reading 
experience. Future challenges include exploring XR applications, refining visual elements 


like color, and addressing issues related to nested graphs in graphical representations. 


Thank you for being a part of this conference. It has been an incredible experience, and 
I appreciate your presence. Now, let's move on to some informal discussions, accompanied 


by refreshments. Cheers! 


Bronnac Ferran 
https://youtu.be/fK1 UObo YONI 
[Transcript in Full] 


I didn't have time to prepare the presentation I wanted, but Frode suggested I share a few 
words. It might have fit better with this morning's presentations. I recently completed a PhD 
on post-war poems and texts in print, titled Auto Poetics. I explored the emergence of non- 
linear and digital tendencies within concrete and visual poetry. I want to reference the first 
theory of text I know, created by Max Benzer in 1957, discussing non-linear texts borrowed 
from Francis Ponge. The text, he argued, could flow under, over, and between things, unlike 
the literary object that elevates things. This concept was presented in 1958 at a visual text 
exhibition in Stuttgart. 

This one. Now, the idea of visual text gained momentum in the 1950s with the rise of 
television and advertising, leading to a linguistic turn in poetry. German post-war language 
works showed a decomposition of the poem into signs, forming semiotic visual poetry. This 
trend continued in the 1960s with a programmatic approach, using the Futura typeface based 
on zero and one lines. Cold type structures emerged, creating textured effects from a single 


letter or two. 


These developments marked a printing and electronic revolution, shifting from 
letterpress to electro photocomposition. My thesis explored such multimedia effects, 
showcasing patterns emerging from a single character, akin to today's AI pattern language 
and attention mechanisms. In 1965, poems began moving off the page into environmental 
contexts, with poem machines and public art projects projecting poetry into various spaces. 

Going back to the theory of texts, scholars like Benzer and Haroldo de Campos traced 
the relational, cyclical text concept back to Mallarmé, Dru Joyce, and others. Benzer, in 1959, 
discussed digital tendencies, emphasizing atomization and fragmentation, resulting in a 
mutable, non-linear text moving beyond the page. 

Reading into the theory reveals a post-war negation of language, creating large blank 


spaces and a depersonalized approach. This contrasts with contemporary generative texts 
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lacking meaning, reminiscent of Chomsky's concerns in the 1960s. 

Leon Van Kammen: Could you elaborate on these poem machines and their impact on 
traditional poetry? 

Bronnac Ferran: Certainly. The poem machines, emerging in the 1960s, marked a 
collective, relational approach among concrete poets. The decentering from the self into auto 
poetics resulted in a collaborative mutation. Changes at a micro-linguistic level, even altering 
one letter, reflected the pattern language seen in today's text generation. The embodied, 
personal relationship in traditional poetry contrasts with the depersonalized, yet relational 
format of generative texts. 

Fabien Benetou: I'm working on a 3D space where text can be manipulated in VR. Could 
this be a solution for exploring generative poetry? 

Bronnac Ferran: Absolutely. The tactile, haptic quality of material poetry, as seen in hot 
metal type, is worth exploring. Engaging with semiotic poetry in a multisensory dimension 
can provide insights for generative text development. 

Frode Hegland: Gesture-based text manipulation, as shown in Fabien's project, might offer 
new possibilities for weaving sentences and exploring generative poetry. 

Bronnac Ferran: Texture is another aspect to consider in this exploration. 

Frode Hegland: Thank you for your insights. If there are no further questions, we'll wrap up 


today's presentations. 
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e Future Text Lab e 
Open Monday Meetings 


We have Future Text Lab meetings every Monday which, if you are interested in the future of 
text, you should feel welcome to join: https://futuretextlab.info 
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Recorded Sessions 


We ran a recent YouTube auto-generated transcript through a https://platform.openai.com 
‘Assistant’. First the transcript was copied and saved as a text file, then the following prompt 
was written in the Assistant and the only user action was then to upload the transcript and 
enter the URL. The sections needed manual line breaks to make more readable and it took a 
while to teach the Assistant to write out the links so that they can work in PDF. Other than 
that, it was all automatic, except for the user having to enter ‘continue’ when the processing 
had not gone through the full document, something further to experiment with. All sessions 
are available on the site or from the website, here is a recent example for how we are 


experimenting with making them accessible: 


Prompt 

You are a creator of YouTube video section headings. I will provide you with a timestamped 
transcript of a YouTube video. You will only be provided with a YouTube URL and a 
document of a full transcript. Please process the full transcript. Please do the following with 
the transcript: Break the video into sections with headings, each followed by a brief 
summary. After each summary add the approximate timestamps where it begins. Use [00:00] 
as the notation for the beginning of the video or [5:30] for 5 minutes and 30 seconds into the 
video, for example. You will be provided with a YouTube URL. Use this URL to create links 
to the video in YouTube based on the timestamps. The links must be written out so that they 
can be read. For example https://youtu.be/HF-QfzEJtqI?si=joxu0HubY GQ9hOq1 &start=0 or 
https://youtu. be/HF-QfzEJtqI?si=joxu0Hub Y GQ9hOq1 &start=809 The participants are likely 
to include: Frode Hegland, Alan Laidlaw, Brandel Zachernuk, Dene Grigar, Fabien Benetou, 
Ismail Serageldin, Leon Van Kammen, Mark Anderson, Peter Wasilko, Rob Swigart & Vint 
Cerf. Other participants sometimes take part. 

This video is a recording of a weekly Future Text Lab meeting. When you have completed, 
pause 10 seconds a continue, making sure you have processed the entire transcript document 


(this did not seem to work, requiring user to add ‘continue’ at regular intervals). 
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13 October 2023 


Understanding Source Abstraction [29:04 - 29:55] 

The conversation delves into the importance of understanding where ideas originate in 
writing and the implications of reading that content through software agents. Mark Anderson 
shares reflections on how abstract representations can impact our perception and 
understanding. Additionally, there is a brief mention of feedback from Ted Nelson, 
highlighting the significance of connectivity in texts. 

Summary Timestamp: [29:04] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=1744 


The Role of AI in Writing [30:10 - 31:03] 

The group touches on the topic of writing with language models and the concept of creating 
"AI clouds" to enhance writing processes. Although the discussion's main proponent, Alan, is 
absent, the idea is teased for future deep dives. 

Summary Timestamp: [30:10] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=1810 


Pioneering Document Manipulation in XR [31:16 - 38:34] 

Fabien Benetou presents his work on manipulating documents within an eXtended Reality 
(XR) setup. He demonstrates how existing software like C web can be a shortcut to managing 
and navigating through complex documents in an immersive environment. The focus shifts to 
exploring user interfaces for interactive documents and how a hybrid approach might allow 
for management of documents in 2D interfaces like smartphones or transition into an XR 
experience. 

Summary Timestamp: [31:16] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=1876 


Expanding Immersive Document Management [38:41 - 39:41] 

Fabien Benetou discusses the potential for immersive document management in XR 
environments, suggesting that not just document reading but also the management of 
document collections, or "shelves," should take advantage of immersive capabilities. He 
emphasizes that the concept extends beyond the final layer of individual document 


interaction to include the broader organizational structures in 3D space. 
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Summary Timestamp: [38:41] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo0552M3a&start=232 1 


Interactive Books in XR and AI Summaries [39:55 - 40:55] 

The group discusses interacting with books in XR and the possibility of linking Al-generated 
summaries with full transcripts. The conversation briefly touches on a system called Vision 
Pro, potentially for virtual experiences, but specific details about it are not publicly disclosed 
during the talk. 

Summary Timestamp: [39:55] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=2395 


Vision Pro and Touch Interactions [40:55 - 43:02] 

Brandel Zachernuk talks about Vision Pro applications, highlighting the default touch 
capabilities that come with system components. The discussion includes ergonomic 
considerations for interaction distances and the potential for getting the application 
experienced on a device through developer evangelism channels. 

Summary Timestamp: [40:55] 

https://youtu. be/HF-QfZEJtqI?si=zN78yL5kzo552MI3a&start=2455 


Incorporating Haptic Feedback in XR Environments [43:36 - 45:00] 

Dene Grigar and Adam discuss the importance of haptic feedback and the visceral 
experiences associated with manipulating physical documents. They consider the challenges 
and opportunities for integrating such sensory inputs into XR environments to emulate the 
engagement with traditional media. 

Summary Timestamp: [43:36] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=2616 


Voice Interaction and Future of UI in XR [45:00 - 47:42] 

Adam talks about the transformative potential of language models in XR interaction, 
particularly for voice commands, which can simplify the user interface by interpreting 
naturally-expressed requests. There is an anticipation of voice becoming a more prominent 
mode of interaction and reducing the number of visible UI elements, allowing intuitive 
commands to manage digital documents. 


Summary Timestamp: [45:00] 
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https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=2700 


Exploring New Ways to Interact with Text in XR [48:00 - 48:31] 

The group explores innovative ways to read text in XR environments, such as opening a 
virtual book with both hands to unfold the text gradually. This approach aims to create a more 
immersive and readable experience. The effectiveness of these interactions is still under 
experimentation, including how it enhances the reading process. 

Summary Timestamp: [48:00] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=2880 


Voice Interaction and Prototyping [48:48 - 50:34] 

Fabien shares anecdotes about his experiences with voice interaction, particularly while 
engaging in activities that preclude typing, like ice skating. He talks about his experimental 
process of using speech to text, text to LLM (language model), and back to speech, 
eventually targeting voice-activated prototyping of software through this iterative 
transformation. 

Summary Timestamp: [48:48] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=2928 


Quest's SP Recognition and Web XR [51:19 - 52:02] 

Brandel discusses the absence of speech recognition APIs in Quest's web platform and the 
importance of this feature, as demonstrated by their presence in Safari and Web XR. He 
expresses his advocacy for these inputs being recognized and integrated by Meta for a richer 
interactive experience. 

Summary Timestamp: [51:19] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo552MI3a&start=3079 


Gestural Interactions in XR and their Impact [52:44 - 53:57] 

The group considers the gap between human-to-human and human-to-computer interaction 
gestures, emphasizing the need for interfaces that allow for more natural and intuitive 
commands, such as using finger pointing as a gestural input rather than relying on persistent 
cursors. 

Summary Timestamp: [52:44] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=3 164 
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Remote Manipulation of Objects in XR [54:48 - 55:50] 


Adam describes the utility of gestural commands like finger pointing in XR environments for 
remote manipulation of objects. He outlines the seamless transition from pointing at an object 


from a distance, to moving it closer, studying it, and then returning it to its original position. 
Summary Timestamp: [54:48] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=3288 


Interaction Dynamics in XR [57:19 - 58:13] 


The discussion turns to interaction dynamics in XR environments. The future of document 
manipulation, both near and far, is envisioned, and the physical desktop space is highlighted 
as a potential variable that could transform virtual interactions with text. Frode envisions a 


phase where virtual tools like pencils could be more realistically used when seated at a desk. 
Summary Timestamp: [57:19] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo552MI3a&start=3439 


Future of Text & Media Synergy [58:28 - 59:10] 


Carrying forward the discussion on XR's future, Frode underscores the significance of the 
interplay between text and other forms of media, especially voice. Acknowledging the 
limitations of each medium when used in isolation, he anticipates an integrated and 


multimodal future for text interaction. 
Summary Timestamp: [58:28] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=3508 


Demonstrating Interaction Feedback Mechanisms in XR [59:10 - 1:00:36] 


Fabien demonstrates a game to showcase interaction feedback mechanisms in XR — the 
fingertips change color to match the target object, providing users with intuitive visual cues. 
This conversation highlights the need for responsive and clear feedback in XR environments 


to aid in task accuracy and user confidence. 
Summary Timestamp: [59:10] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=3550 


Text Annotation and Engagement [1:04:06 - 1:05:27] 


Peter Wasilko discusses the idea of reading and engaging with novels through digital means, 
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which allows for annotations without the "desecration" associated with physically marking a 
book. This leads to a larger discussion on how digital tools might allow every reader to 
engage with and personalize their experience of a text. 

Summary Timestamp: [1:04:06] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=3846 


Distinguishing Original Text from Personal Annotations [1:05:27 - 1:05:59] 

The conversation emphasizes the importance of distinguishing between the original text of a 
book and the reader's personal annotations. This distinction is crucial for curating a personal 
experience with a novel while preserving the integrity of the original text. 

Summary Timestamp: [1:05:27] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo552MI3a&start=3927 


Exploring Document Tables of Contents [1:06:13 - 1:09:17] 

Frode shares his reflections on the inconvenience of static tables of contents for extensive 
documents. He presents an approach that employs color-coding and author name indexing to 
create a more skimmable content structure. Frode reveals how removing the title from the 
table of contents can reduce distraction and improve informational context. 

Summary Timestamp: [1:06:13] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=3973 


The Necessity of Real Headset Experience [1:09:17 - 1:10:21] 

The conversation urges participants like Peter to experience modern XR headsets to truly 
understand the distinctiveness of interaction within these environments. Frode suggests that 
the actual use of an XR headset can enhance one's perspective on the spatial and gestural 
dynamics of virtual spaces. 

Summary Timestamp: [1:09:17] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=4157 


Remarks on Future XR Headsets and Web Interactivity [1:10:21 - 1:11:25] 

Fabien comments on the anticipated arrival of a new headset from a French company, 
highlighting its ease of use without the need for accounts, and discusses the interaction 
mechanisms relevant even without an XR device. Brandel addresses web interactivity and its 


potential improvements. 
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Summary Timestamp: [1:10:21] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo552MI3a&start=422 | 


Innovative Page Navigation in XR [1:11:25 - 1:14:45] 

Fabien demonstrates an innovative gesture for navigating through pages in an XR 
environment by stretching a widget or text to select a number of pages to move through. This 
idea explores the translation of physical gestures into digital commands in an immersive 
context, and the group reflects on the use of spirals or log scales for navigation. 

Summary Timestamp: [1:11:25] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=4285 

This comprehensive set of sections provides a framework to navigate the YouTube video's 
content, offering a nuanced look into the frontiers of text and technology as explored by the 
Future Text Lab meeting. The summaries and corresponding timestamps serve as a guide to 


the rich dialogue and conceptual explorations discussed in the meeting. 


Fine and Large Movement Dynamics in XR [1:14:53 - 1:16:09] 

Brandel brings up the relationship between small and large movements in XR interactions, 
explaining the need for fine control adjustments following gross translations. Emphasizing 
the ease and recoverability of such interactions, he underscores the importance of scaling 
movements for better positioning in XR. 

Summary Timestamp: [1:14:53] 

https://youtu. be/HF-QfzZEJtqI?sizN78yL5kzo552MI3a&start=4493 


Potential Contribution to the Book [1:16:09 - 1:18:14] 


Frode suggests Brandel write about the interaction dynamics for the Future of Text book, 
stressing the significance of his insights. Brandel expresses willingness to share and discusses 
the opportunity of speaking at a Carnegie Mellon panel to further disseminate best practices 
for the web and XR. 


Summary Timestamp: [1:16:09] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=4569 


Exploring Non-Euclidean Representations [1:18:14 - 1:21:55] 


Leon opens up a discussion on the conceptual difference between thinking and actually being 


in XR, presenting a demo that uses non-Euclidean spaces and portals. He explores the idea 
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that links can be imagined as non-physical gateways moving away from traditional spatial 
constraints and into a new dimension of document navigation. 

Summary Timestamp: [1:18:14] 

https://youtu. be/HF-QfZEJtqI?si=N78yL5kzo552MI3a&start=4694 


The Cognitive Science of Navigating Non-Euclidean Spaces [1:21:55 - 1:23:58] 


Fabien chimes in to discuss the cognitive aspects behind navigating non-Euclidean spaces 
within games and XR, suggesting that our understanding of spatial relationships can adapt to 
these new modalities. He also highlights the limitations of Euclidean arrangements for 


document management in XR and the need to onboard users to new spatial efficiencies. 
Summary Timestamp: [1:21:55] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=49 15 


Reflection on XR Interventions and Impact [1:40:46 - 1:42:00] 
Brandel talks about the quality and potential impact of his XR projects and emphasizes that 


even small, seemingly haphazard efforts can substantially influence the broader industry's 
understanding and advancement. He suggests that personal contributions can have a 


surprisingly large effect on the field. 
Summary Timestamp: [1:40:46] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=6046 


Contemplation on Personal Relevance and Contributions [1:42:00 - 1:43:14] 


Frode shares a reflective moment of self-assessment, discussing emotional oscillations 
between feeling the significance of one's work and questioning its value. A personal anecdote 
from a Rome conference offers a gratifying perspective that even small contributions can 


influence future generations. 
Summary Timestamp: [1:42:00] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=6 120 


Embracing Interactive Possibilities In Authoring [1:43:14 - 1:45:38] 


An interactive discussion about the possibilities of linking within the context of text 
authoring transpires, underscoring the enhanced engagement that comes with the ability to 


navigate and explore connected ideas dynamically. 


Summary Timestamp: [1:43:14] 
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https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552MI3a&start=6 194 


Hyperlinking Dynamics and History [1:45:38 - 1:46:11] 
Mark Anderson raises the notion of hyperlinking and its historical significance, leading to a 


conversation about the application and presence of these features in various platforms beyond 
the web, such as Adobe Acrobat and Microsoft Word. 


Summary Timestamp: [1:45:38] 
https://youtu. be/HF-QfzEJtqI?si=N78yL5kz0552M3a&start=6338 


The Role of PDF and Its Potential Evolution [1:46:11 - 1:47:58] 

Mark continues to ponder the historical purpose of PDF for rendering consistency across 
printers and its contemporary usage. He advocates for an evolution of the format—or an 
alternative—towards structures that provide richer affordances for multidimensional 
interaction. 

Summary Timestamp: [1:46:11] 

https://youtu. be/HF-QfzEJtqI?si=N78yL5kzo0552M3a&start=637 1 
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e The History of Text Timeline 


History of Text Timeline 


This will never be, nor aims to be, a complete and accurate history of text. There will be 
errors in omission, facts and dates will only be solid for the most recent events. The timeline 
format is ill suited for non-sharply delineated periods of time so we have tried to address that 
with language, such as liberal use of ‘ca’ and date ranges. The history of ideas is especially 
fraught and there will be issues we have not even thought about. What this aims to be 
however, is a useful guide for at least some of the major events and sequences which has 
brought us where we are, and which may help guide us to where we want to be with text. 
Since the format is so simple we aim that it should at least be useful to students to get a lay of 
the temporal land. 

For any suggestions or issues, please email the editor Frode Alexander Hegland at 
frode@hegland.com and you will be credited as a Contributor in the next Edition. It would be 
great if you could use this format: Year (even if you have to use ‘ca’ or other terms) Event/ 


thing by person at organisation (if applicable) 
13,8 Billion Years Ago 


ca 13,800,000,000 years ago the universe comes into being. There was no ‘instant’ of 
creation. The universe didn’t flash into existence, it came into being as an all-encompassing, 
interactive, quantum wave. There is no going back. From pure energy to all there is today, the 


universe gets more complicated and more interactive one Planck moment at a time 
ca 4,540,000,000 years ago earth and the solar system is formed 


ca 4,400,000,000 years ago earth’s oceans form, providing a substrate for life with rich 


potential for interactions 


Let’s pause before we continue the journey into the next, great step (that of life itself). Look 
at these dates - the solar system has been around for roughly 1 of the universe’s existence. 


That is something to marvel at. It’s easy to imagine vast intergalactic civilizations having 
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come and gone over the life of the universe, but it turns out that there actually isn’t that much 
time in the past. We’re pretty early inhabitants. There may have been one generation of stars 
similar to our own before us —maximum. So, maybe there hasn’t been enough time for 
advanced civilizations to evolve. That we might be one of the most advanced consciousness 


in creation (or perhaps the only one) is a sobering thought. Can we handle this responsibility? 


ca 4,000,000,000 Self-replicating molecules appear. Life is happening. It’s pretty basic, but 
it’s happening 

ca 3,500,000,000 Single-celled organisms 

ca 3,000,000,000 Viruses, though they may be much older 

ca 580,000,000 Complex multicellular life 

ca 250,000,000 or less—it is hard to be sure, DNA, with complex ‘letters’ of interaction takes 


life to a whole new level 


250 Million-3,6 Million 


2.7-2.5, 1.9-1.7 and 1.1-0.9 million years ago, the earth sees rapid climate change (on the 
scale of lifetimes of individuals, not species) spurring on hominid evolution in the Rift Valley 
in Africa, with each period coinciding with brain development. During the period 1.9-1.7 the 
number of hominid species reached its peak and Homo Erectus appeared. Tool development 
also coincided with these cycles of rapid climate change, including Oldowan, Acheulean and 
Mousterian. For more on this topic, and how the planet shaped us in general refer to Origins 


by Lewis Dartnell 
ca 3,600,000 Our ancestors walk upright and they loose body hair 
ca 2,300,000 Homo Habilis, the tool user, is our oldest ancestor to use tools 


ca 2,000,000 Olduwan tool Culture begins. Its key feature was the method of chipping stones 


to create a chopping or cutting edge. 


2,000,000-50,000 BCE 


ca 500,000 Earliest evidence of purpose-built shelters, found near Chichibu, Japan 
ca 400,000 Early humans begin to hunt with spears 


ca 280,000 First complex stone blades and grinding stones 
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ca 150,000 Humans possibly capable of speech 
ca 100,000-200,000 Modern Humans 


50,000-3,000 BCE 


ca 50000 Our ‘Great leap forward’. Human culture starts to change more rapidly (burying our 


dead ritually, clothes from animal hides, complex hunting techniques) 


ca 44000 Oldest known cave painting, found in the Franco-Cantabrian region in western 


Europe and Sulawesi, Indonesia 
ca 35400 Oldest-known example of figurative art, in Sulawesi, Indonesia 
ca 11000 Cave art by young children in the Rouffignac Cave 


ca 7500 Near Eastern counters “Tokens’ to keep track of goods are the earliest known 


antecedents of the Mesopotamian Cuneiform script 


ca 6600 Eleven isolated symbols carved on tortoise shells were found at Jiahu, an 
archaeological site in the Henan province of China, some bearing a striking resemblance to 


certain modern characters but the connection is not established 


ca 4500 Proto-Indo-European language developed, probably somewhere near the Black Sea, 
and probably spreading because its speakers invented horse riding. Today 60% of modern 


humans speak a daughter language, 27% as their mother tongue 


ca 4000 Possible preliterate images which may have been symbols (such as Gerzean pottery) 
which could have been precursors to Egyptian hieroglyphic writing 

ca 3500 Egyptian Proto-hieroglyphic symbol systems 

ca 3300 Reduction of three-dimensional Near Eastern tokens into two-dimensional signs on 
envelopes holding tokens 


ca 3200 First logographic Near Eastern accounting lists written on clay tablets by impressing 


tokens 
ca 3100 First logographic proto-cuneiform signs traced with a stylus on accounting tablets 


ca 3000 First proto-cuneiform phonetic signs to represent personal names on economic 
tablets 


ca 3000 First known use of papyrus for writing. Previously Egyptians had been writing on 


stone and pottery 


ca 3000-1000 Hieratic (‘priestly') cursive writing system used for Egyptian until the rise of 


Demotic. Primarily written in ink with a reed pen on papyrus. 
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3000 BCE 


2900 First known air mail. Egyptian sailors released carrier pigeons from ships to pre- 


announce their arrival 


ca 2800 First full sentence written in mature Egyptian hieroglyphs so far discovered. Found 


on a seal impression in the tomb of Seth-Peribsen at Umm el-Qa'ab 

ca 2700 First cuneiform texts which departs from accounting: funerary texts 
ca 2600 Sumerian language develops 

ca 2600 Egyptian language develops 

ca 2400 Akkadian language develops 

ca 2400 First cuneiform tablet dealing with trade 


ca 2300 First written sentences. These texts were inscribed on worshippers’ votive statues 


dedicated to a god and requesting immortality 
ca 2300 First named author, Enheduanna, daughter of Sargon the Great 


ca 2300 Oldest known dictionaries of cuneiform tablets with bilingual Sumerian—Akkadian 


wordlists, discovered in Ebla (modern Syria) 
ca 2000 Classical period of the Sumerian Cuneiform Script 
ca 2000 First known library catalog in the Sumerian city of Nippur 


ca 2000 Abacus (from Greek meaning “board strewn with sand or dust used for drawing 


geometric figures or calculating”), the first known calculator, is invented in Babylonia (Iraq) 
ca 2100 Elamite language develops 


ca 2100-1500 Proto-Sinaitic script, the earliest trace of alphabetic writing known, in the 


Egyptian Pharaoh’s turquoise mines at Serabit el-Khadim in the Sinai Peninsula 


2000 BCE 


ca 1900 First known cipher (not yet decoded), in tomb of Khnumhotep IT 

ca 1750 Hammurabi’s Code, by Hammurabi, ruler of Babylon 

ca 1700 Hittite language develops 

ca 1600 Earliest known medical document, the Edwin Smith Medical Papyrus, thought based 


on material from 3000 BCE, including the first reference to the human brain 


ca 1500 Phoenician alphabet of 22 consonants was among the early mature alphabets. It 


spread over the Mediterranean and led to the Greek, Hebrew, Roman, Arabic and modern 
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alphabets 
ca 1450 Greek language develops 
ca 1500 Earliest book known, the Ebers papyrus, a 20 meter scroll 


ca 1500 First known use of movable type (stamps reused to repeat symbols identically), the 
Phaistos Disc, and first font 


ca 1300 First known inclusion of words on a map, in Mesopotamia 


ca. 1300-1190 The Ugaritic writing system a cuneiform augmented abjad (consonantal 


alphabet) for Ugaritic, an extinct Northwest Semitic language 


ca 1300s Wax tablet with stylus: origins are uncertain but known to have been used at least 


until the 1860s CE, for example in the fish market in Rouen, France 

1200s Late Bronze Age collapse 

ca 1250-1192 Earliest confirmed evidence of Chinese script, Oracle bones script 

ca 1200 Torah was copied onto a scroll by Moses according to the Hebrew tradition (date 
disputed) 

ca 1200 Old Chinese language develops 

ca 1100 BC — 256 Chinese Jinwen (Bronzeware Script) 

1000-300 Chinese bronze inscriptions/script 

1000s the Gezer Calendar, first vertically-formatted list 


ca 1000 Hebrew language develops 


1000 BCE 


1000 Chinese Seal script evolved organically out of the bronze script 

900-400 The Greek Alphabet emerged around the ninth or eight century BCE which had 
distinct letters for vowels, not only consonants. Many versions of the Greek alphabet existed 
but by the fourth century it had been standardised into twenty-four letters, ordered from alpha 
to omega 

ca 700 Latin language develops 

700s Alphabetic writing entered the Greek world from the Levant 

650-400s CE Demotic Egyptian script following Late Egyptian and preceding Coptic. The 


term was first used by the Greek historian Herodotus to distinguish it from hieratic and 
hieroglyphic scripts 


500s First known curated museum. Mesopotamian artifacts spanning 1,500 years, by Princess 
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Ennigaldi, daughter of King Nabonidus 
ca 500 Sanskrit language develops 
ca 550 First official mail service, by Cyrus the Great, stretching from Post, Iran to Hakha, 


Myanmar 

ca 500 Astadhyayt by Panini, quasi-generative grammar of Sanskrit, anticipating Chomsky 
300s The basic form of the Codex invented in Pergamon 

ca 300 Tamil language develops 

300s Reed pens for writing on papyrus 

310/305—240 The Pinakes, the first library catalog at the Library of Alexandria 

285-246 Alexandria founded by Alexander the Great 

283 Library of Alexandria founded by Ptolemy I and II 

257-180 Punctuation is invented at the Library of Alexandria by Aristophanes of Byzantium 
256-206 Chinese Zhuanshu (Seal Script). 

206 BC—220 AD Chinese Zhuanshu simplified to Lishu (Clerical script) 

250 Parchment Scrolls 

ca. 230 The letter ‘G’, by Spurius Carvilius Ruga, the first known inventor of a letter 

200s Quill used until about the 19th century CE, when replaced by the pen 

200s Alphabetization developed, probably in Alexandria by Callimachus to catalog the Great 
Library 

200s Erya, first known dictionary 

ca131-59 Acta diurna, daily news by government, published in Rome 

179-141 Earliest extant paper fragment in Fangmatan in Gansu province, China 


before 134 First character encoding, by Cleoxenus and Democleitus, described by Polybius. 


Each Greek letter was converted to 2 digits (1 to 5), then to smoke or fire signals 
63 & ‘ampersand’ proposed by Marcus Tiro 
ca 55 The book in the form of folded sheets, not just a stack of sheets, by Julius Caesar, in his 


reports on the Gallic Wars 


S27 


BCE — CE 


ca 50 Earliest surviving example of Old Roman Cursive script: a speech by Claudius 
79 Earliest tables of contents by Pliny the Elder in Naturalis Historia (Natural History) 
79 Earliest known marketing pun and portmanteau word: wine jars in Pompeii marked 
‘Vesuvinum’ (Vesuvius wine) 

79 Two SATOR AREPO word squares in Pompeii, perhaps with Christian associations, 


making them the earliest surviving Christian inscriptions 


100 CE 


200 


ca 200 New Roman (or Minuscule) Cursive script which evolved into modern lower case 


letterforms 


ca 220 Earliest surviving woodblock printed fragment (China) 


300 


ca 300 Maya writing 

ca 300 Latin handwriting starts to use larger letters at the start of sentences, though the same 
shape (not mixed case) 

330-360 Codex Sinaiticus, the oldest extant codex; a biblical manuscript written in Greek 


367 Old Roman Cursive script banned except for official imperial documents, eventually 
leading to lower case text (derived from New Roman Cursive) being normal and upper case 


exceptional 


400 


420-589 Chinese Kaishu script (Regular Script) replaces Lishu 
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500 


Before 500s Literacy introduced to Japan in the form of the Chinese writing system, via 
Baekje 

500-1000 Florilegium, which are selections of ‘flowers’(select passages) from work, rather 
than a summary, to help people deal with the volume of books 


593 Woodblock printing starts in China 


600 


600s Quill pens, made from the outer feathers of crows and other large birds, becomes 


popular 


700 


ca 700s Word spacing pioneered by Celtic monks 


ca 700 St Cuthbert Gospel, the oldest surviving Western book, which still has its original 
goatskin leather cover 


700s Japanese writing develops away from Chinese 


764 Empress Koken commissions the earliest known examples of woodblock printing in 


Japan 


800 


800s paper starts to replace parchment as the primary writing material for administrative uses 
in Baghdad 


813 Council of Tours decreed sermons should be in vulgar language not Latin. This may have 
triggered early Romance languages to be spelt literally, rather than as Latin with distorted 


pronunciation 


842 Oaths of Strasbourg, first surviving document in Romance (early French), with parallel 


version in Frankish (early Germanic) 
868 The oldest known printed book, The Diamond Sutra, a Buddhist book in China 


854-931 Prototype professional peer-review process recommended in the Ethics of the 
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Physician written by Ishaq ibn ‘Alt al-Ruhawt 


900 


ca 900 Screen Printing in China during the Song Dynasty 


900s Arabic numerals come to Spain, though they were not commonly used until the 


fourteenth century. 


960-1279 Chinese Kaishu script evolves to Songti script 


1000 


1080 The Missal of Silos, the oldest known document on paper created in Europe 


1056 First recorded paper mill in Xativa on the Iberian Peninsula 


1100 


1190 First paper mill in France 


1200 


1200s The term ‘Originalia’ is coined in contrast to Florilegia, indicating a greater authority 


to original sources than excerpts 


1246 Call numbers associated with the location of books, in the Library at Amiens Cathedral 


in France 
1276 Paper mills established in Italy 
1290 Ars Magna by Ramon Lul 


1300 


1377 Jikji the oldest surviving book printed using moveable metal type by Gyeonghan in 
Korea 


1300s The word ‘history’ meant, “relation of incidents whether true of false.” The word goes 
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back to the Proto-Indo-European root of wid-tor weid, it literally mean “to know” and “to 
see.” 

1304-1374 Humanism founded by Francesco Petrarch, reviving enthusiasm for ancient 
Roman thinkers, with books as the centre of their discourse 

1320 First paper mills in Germany 

1340-1350 First paper mills in Holland 


1346 First known two-color print, a frontispiece to a Buddhist sutra scroll 


1400 


1400s First prototype of a Jacquard-type loom by Jean le Calabrais 

1424 The University of Cambridge has one of the largest libraries in Europe with just 122 
books. Books are still handwritten on parchment 

1453 Constantinople captured by the Turks and books from its Imperial Library are burned or 
removed, marking the end of the last of the great libraries of the ancient world 

1455 ‘Gutenberg Bible’, also-called Forty-two-line Bible, or Mazarin Bible, the first 
complete moveable type printed book extant in the West, printed by Johannes Gutenberg 
1457 First known color printing is used in Mainz Psalter by Johann Faust and his son-in-law 
Peter Schéffer 

1470 Roman typeface, the first recognisably modern typeface, a combination of capital letters 
inspired by ancient Roman architectural inscriptions and Carolingian minuscules, developed 
by Nicolas Jenson 

1470 First printed joke book, Facetiae by Poggio Bracciolini 

1470 Earliest extant example of sequential numbering in a book, Sermo in festo 
praesentationis beatissimae Mariae virginis, printed in Cologne. This did not become 
standard for another half century. Peter Schoffér, apprentice of Gutenberg, is the inventor of 
the title page and Arnold Therhoernen in Cologne, is one of the first to use both a title page 
and page numbers 

Late 1470s, title, author, and publisher information included by printers on the first inside 
page of a book 

1479 Manicule in Breviarium totius juris canonici, compiled by Paolo Attavanti printed in 
Milan by the German firm of Leonhard Pachel and Ulrich Scinzenzeller 

1481 First marginal annotations used in printed texts on a Venetian edition of Horace with 


commentaries by Acro and Porphyry 


23) 


1483 First Talmud printed 
End of the 1400s almost all printed books have title pages 
End of the 1400s the numerals 4, 5, and 7 begin to take the forms we are familiar with today 


1500 


1500-1700 Handwrittern newsletters in Europe called avvisi, reporti, gazzette, ragguagli, 


nouvelles, advis, corantos, courantes and zeitungen 


1500s Garamond typeface. Claude Garamont, a French type designer, publisher and punch- 
cutter lived in Paris. Thus, many old-style serif typefaces are collectively known by his name 
as ‘Garamond’ 

1500s The word ‘history’ is differentiated into ‘history’ and ’story’ in English, though in other 
languages, such as Spanish and Norwegian there is still no distinction 

1500s Maya writing mostly fallen out of use 

ca1500 Etching for printing by Daniel Hopfer 

1501 Italic typeface by Aldus Manutius 

1513 Likely first pagination with Arabic numerals in Cornucopiae by Niccoloo Perotti 


1517 Martin Luther posts a thesis against indulgences and thus sparking what would be 
called the Reformation, a questioning of authority which would spur greater literacy rates and 


interest in education 

1530s Monasteries disolved in England 

1538 Latin-English wordbook by Sir Thomas Elyot 

1539 Henry the Eighth’s Great Bible, by Myles Coverdale banning all glossing 


1540 Henry the Eighth’s authorised Grammar, which formed the basis of schoolbooks in 
England for the next 300 years 


1545 Bibliotheca universalis by Conrad Gessner, a complete bibliography of all printed 
books (except itself) 


1556 Notizie Scritte, first monthly newspaper published in Venice 


1557 The Geneva Bible, the primary Bible of 16th-century English Protestantism displaces 
the Great Bible 


1560 First blueprints for the modern, wood-encased carpentry pencil by Simonio and 


Lyndiana Bernacotti 


1564 Graphite for pencils comes into widespread use following the discovery of a large 
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graphite deposit in Borrowdale, England 


1568 Bishops' Bible, English translation of the Bible produced under the authority of the 
established Church of England and later used as the base text for the King James Bible 


1575 First paper mills in Mexico 

1565 Mechanical/Lead holder pencil by Conrad Gesner 

1588 First commercially successful paper mill in Britain by John Spilman in Kent 
1593 Index to content in a book, by Christopher Marlowe in Hero and Leander 


1595 The first printed catalog of an institutional library, the Nomenclator of Leiden 
University Library 


1600 


1600 Orbis Sensualium Pictus textbook for children by John Amos Comenius 


1604 Relation aller Fiirnemmen und gedenckwiirdigen Historien, first weekly newspaper, 


published in Germany by Johann Carolus 

1611 King James Bible 

1642 Mezzotint Printmaking by Ludwig von Siegen 

1648 Part emoticon ‘(smiling yet:)’ by poet Robert Herrick 

1665 Journal des sgavans, in Paris, first academic journal 

1665 Philosophical Transactions of The Royal Society, in London, second academic journal 
1665 Oxford Gazette, first English newspaper 

1667 Acoustic string telephone by Robert Hooke 

1674 First decipherment of a script, the Staveless Runes, by Magnus Celsius 
1677 Artificial versifying by John Peter 

1600s Quills become more pointed and flexible 

1690 First paper mills in the USA 


1700 


1702 The Daily Courant, the world's first daily newspaper, printed on paper so cheap it was 


designed to be thrown away after reading 


1704 Daniel Defoe, considered the first journalist, publishes The Review 
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1704 Newton’s Opticks, the first major scientific book published in English, not Latin 
1706 Newton’s Opticks translated into Latin 

1714 First patent for a mechanical typewriter issued to Henry Mill 

1723 De Etruria regali libri VII Thomas Dempster used sans serif typeface to represent 


inscriptions in Ancient Greek and Etruscan 


1725 Improvement to the Jacquard-type loom by Basile Bouchon who introduced the 
principle of using a perforated band of paper 

1731 First peer-reviewed journal, Medical Essays and Observations (Philosophical Society of 
Edinburgh, Edinburgh). 

1739 Last international treaty written in Latin, the Treaty of Belgrade, indicating the new pre- 
eminence of living languages over dead ones 

1748 First modern use of sans-serif (““grotesque”’) lettering, anonymous letter carver, grotto at 
Stourhead, England 

1755 A Dictionary of the English Language by Samuel Johnson 

1767 Index Card organization by Carl Linnaeus 


1769 Every house in Britain needs to have a number for addressing, introduced with the 
Stamp Act, 


1770 Natural rubber used as an eraser by Edward Nairne 

1771 UK Parliament formally gives journalists the right to report proceedings 

1772 Aquatint printing by Peter Perez Burdett, named by Paul Sandby 

1780 Didot and Bodoni by Firmin Didot and Giambattista Bodoni, the first modern modern 
Roman typefaces 

1780 First card catalog by librarian Gottfried van Swieten, Prefect of the Imperial Library, 
Austria 

1783 James Madison of Virginia proposes the creation of a congressional library 

1786 Rounded sans-serif script font developed by Valentin Hatiy for the use of the blind to 
read with their fingers 

1787 Constitution of the United States, mentioned here as a milestone in written documents 
producing and framing a society 

1787 The Federalist Papers by Alexander Hamilton with John Jay and James Madison in The 
Independent Journal, considered the most important documents for interpreting and 
understanding the original intent of the Constitution of the United States 

1791 First card catalog for libraries, using the back of playing cards by a group of men with 
bibliographic experience led by Barthélemy Mercier 
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1795 Modern Pencil by Nicholas-Jacques Conté 
1796 Lithography by Alois Senefelder 
1796 Colour Lithography by Alois Senefelder 


1799 The Fourdrinier machine, a continuous paper making machine by Louis-Nicolas Robert 


of France 


1800 


1800 The Library of Congress established when President John Adams signed an act of 
Congress also providing for the transfer of the seat of government from Philadelphia to the 


new capital city of Washington 

1801 Blackboard by James Pillans 

1801 Carbon Paper by Pellegrino Turri 

1804 Jacquard loom by Joseph Marie Jacquard 
1806 Patent for Carbon Paper by Ralph Wedgwood 


1810 


1816 First typeface without serifs by William Caslon IV 
1816 First working Telegraph by Francis Ronalds used static electricity; it was rejected by the 


Admiralty as “wholly unnecessary” 


1817 A Code of Signals for the Merchant Service, the first general system of signalling for 


merchant vessels by Captain Frederick Marryat 


1819 Rotary printing press by David Napier 


1820 


1822 Mechanical Pencil with a ‘Mechanism to Propel Replaceable Lead’ by Sampson 


Mordan and John Isaac Hawkins 
1828 Pencil Sharpener by Bernard Lassimonne 


1829 Embossed printing invented by Louis Braille 
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1830 


1836 Chorded Keyboard by Wheatstone and Cooke 
1837 Early forerunner of Morse Code by Samuel F. B. Morse, Joseph Henry, and Alfred Vail 
1838 Stereopsis described by Sir Charles Wheatstone, leading him to construct the 


stereoscope 
1839 Vulcanized rubber used for erasers by Charles Goodyear 
1839 Electrical Telegraph commercialised by Sir William Fothergill Cooke 


1840 


1843 Rotary Drum Printing by Richard March Hoe 
1843 Wood pulp introduced to paper mills for paper production 
1844 Newsprint by Charles Fenerty of Canada. Designed for use in printing presses that 


employ a long web (continuous sheet) of paper rather than individual sheets of paper 
1844 Morse Code by Samuel F. B. Morse, Joseph Henry, and Alfred Vail, in use 
1846 Printed Output envisioned by Charles Babbage from his Difference Engine 2 


1850 


1850s Appearance of middle class women magazines UK 

1854 Boolean algebra the mathematical basis of digital computing, developed by George 
Boole in The Laws of Thought 

1855 International Code of Signals drafted by the British Board of Trade 

1857 International Code of Signals published as the Commercial Code 

1857 National Telegraphic Review and Operators Guide lists emoticon precursors <3 and :* 
as shorthand for ‘love and kisses’ 

1857 Study On Some Deficiencies in our English Dictionaries, which identified seven distinct 
shortcomings in contemporary dictionaries published by the Unregistered Words Committee 
of The Philological Society, a small group of intellectuals in London headed by Richard 
Chenevix Trench 

1858 Eraser on pencil by Hymen Lipman 

1858 First transatlantic telegraph cable laid by Cyrus West Field 
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1860 


1860s The first card catalog, designed for readers, rather than staff, by Ezra Abbott, Harvard’s 


assistant librarian 

1860 Herbert Coleridge succeeds Richard Chenevix Trench as the first editor of the 
Unregistered Words Committee’s effort; this work was the precursor of what eventually 
became the Oxford English Dictionary (OED) 

1860 Hectograph, gelatin duplicator or jellygraph printing process by Nelson S. Knaggs 

1860 The New York Herald starts the first ‘morgue’, meaning archive 

1861 The Unregistered Words Committee published the first sample pages, Herbert Coleridge 


dies and Frederick Furnivall takes over as editor 

1864 Non-Digital ‘spam’. Unsolicited group telegram advertisement 

1868 Kineograph / Flip-Book by John Barnes Linnett 

1868 The Remington by Christopher Latham Sholes, the first successful typewriter 


1870 


1870s QWERTY layout by Christopher Latham Sholes 
1874 Stencil Duplicating by Eugenio de Zuccato 

1875 First literary agents 

1876 Telephone patent by Alexander Graham Bell 


1876 Telephone Switch, which allowed for the formation of telephone exchanges and 
eventually networks by Tivadar Puska 

1876 Autographic Printing by Thomas Edison 

1879 The Oxford University Press agrees to publish The Unregistered Words Committee’s 
dictionary, to be edited by James Murray 


1879 Index Medicus edited by John S. Billings and Robert Fletcher, published by Frederick 
Leypoldt 
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1880 


1828 On the recent Improvements in the Art of Printing published in The Quarterly Journal 
of Science, Literature, and Art, by Edward Cowper 


1850 On Printing Machines, Especially Those Used in Printing 'The Times' Newspaper 
published in Institution of Civil Engineers. Minutes of Proceedings, by Edward Cowper, 
outlining his contribution to printing which had increased newspaper printing from 200-250 
copies per hour on a hand press to 10,000 copies per hour 

1873 First illustrated daily newspaper, The Daily Graphic, published in New York. 

1877 Current definition of entropy, by Ludwig Eduard Boltzmann 

1881 Harvard Citation Style (author date) by Edward Laurens Mark at Harvard University 


1881 Emoticon precursors as Puck magazine published a set of type-set faces expressing joy, 


melancholy, indifference and astonishment using basic type characters 
1883 Téléphonoscope concept by Albert Robida 
1884 Linotype by Ottmar Mergenthaler 


1884 The Oxford University Press agrees to publish A New English Dictionary on Historical 
Principles; Founded Mainly on the Materials Collected by The Philological Society 


1887 Snigger Point by Ambrose Bierce, a precursor emoji/emoticon symbol in the form of an 


opening parenthesis character ‘(’, but rotated 90° to the left 


1888 Ballpoint Pen by John J. Loud 


1890 


By 1890 Some papers boasted circulations of more than one million 

1890 US Census undertaken using the punched-card technology, an invention suggested by 
John S. Billings to Herman Hollerith in the company which would become IBM 

1891 Prototype of the Kinetoscope, which enabled one person at a time to view moving 
pictures by the Edison Company 

1891 Automatic Cyclostyle duplicating machine by David Gestetner 

1895 Universal Decimal Classification (UDC), starting with the Universal Bibliographic 
Repertory (RBU: Répertoire Bibliographique Universel) by Paul Otlet and Henri La Fontaine 
with the implementation being as card catalogue by Herbert Haviland Field, using the Dewey 
Decimal Classification system by Melvil Dewey 


1894 Information and Entropy in Thermodynamics by Ludwig Boltzmann 
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1895 A New English Dictionary on Historical Principles renamed as the Oxford English 
Dictionary (OED) 


1900 


1901 Trans-Atlantic Radio Signal by Marconi Company 

1902 The term Diglossia coined by Karl Krumbacher to refer to the phenomenon of 
divergence between spoken and written language 

1903 First message to travel around the globe by Commercial Pacific Cable Company, from 
US President Theodore Roosevelt, wishing “‘a happy Independence Day to the US, its 
territories and properties...” It took nine minutes for the message to travel worldwide 

1903 The Daily Mirror, the first tabloid-style newspaper 

1904 Patent for a ‘type wheel printing telegraph machine’ filed by Charles Krum which 
would go on to become Teletype in 1929 

1906 British Kinemacolor process for natural film colour 

1906—7 Photographic Copying Machines by George C. Beidler at the Rectigraph Company 
1907 Commercial Transatlantic Radio Telegraph Cable opened by Marconi Company 


1910 


1910 Felt-tip marking pen by Lee Newman 

1910 Mundaneum by Paul Otlet and Henri La Fontaine 

1910 First criminal caught via wireless telegraph: the murderer Dr Crippen on board a 
transatlantic ship 

1913 Plantin typeface by Frank Hinman Pierpont and draughtsman Fritz Stelzer of the British 
Monotype Corporation, based on a Gros Cicero face cut in the 16th century by Robert 
Granjon 

1914 Optophone (OCR precursor) by Emanuel Goldberg, a machine which read characters 
and converted them into standard telegraph code 


1914 Handheld Scanner (OCR precursor) by Edmund Fournier d’ Albe a machine which read 


characters and converted them into tones 
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1920 


1920s First full-time Type Designer Frederic Goudy 


1922 Ulysses by James Joyce, first extensive use of stream of consciousness: text conveying 


thoughts not speech 


1923 Spirit duplicator (also referred to as a Ditto machine, Banda machine, or Roneo) by 
Wilhelm Ritzerfeld 


1924 Art Color Pencils by Faber-Castell and Caran d’Ache 
1925 Corkboard by George Brooks 
1926 Information in physics by Leo Szilard 


1926 research and development which would become Telex initiated by Reichspost in 


Germany 

1927 The Statistical Machine patented by Emanuel Goldberg 

1927 First film with audio, The Jazz Singer, using Warner Brothers’ Vitaphone system 
1927 Futura typeface family by Paul Renner 

1928 Standardised punch cards by Clair D. Lake 

1929 Hellschreiber by Rudolf Hell, precursor to dot matrix printing 

1929 Practical Criticism: A Study of Literary Judgment by Richards, L.A 


1930 


1930 The Readies, a concept for portable speed reading by Bob Brown 

1931 Knowledge Machine by Emanuel Goldberg 

1931 Biro by brothers Laszl6 Biré and Gyorgy Bird 

1931 The American Telephone and Telegraph Company (AT&T) introduced its teletypewriter 


exchange service, TWX 


1932 Times New Roman typeface by Victor Lardent under the direction of Stanley Morison, 


on a commission of the Times newspaper, based on the Plantin typeface 
1932 Information in quantum and particle physics by John von Neumann 
1933 Telex by Reichspost in Germany operational 

1933 Machine translation by Petr Petrovitch Smirnov-Troyanski 


1934 Logik der Forschung by Karl R. Popper advanced the theory that the demarcation of the 


limit of scientific knowledge, is its ‘falsifiability’ and not its ‘verifiability’ 
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1934 Mundaneum/ “Mondothéque,” by Paul Otlet. Includes automated linking between “card 
catalogs with sixteen million entries, photos, documents, microfilm, and more. Work on 
integrating telegraphy and multiple media, from sound recordings to television” 

1935 Fictional model for VR by science fiction writer Stanley Weinbaum in his short story 
Pygmalion's Spectacles 

1935 Monde book by Paul Otlet 

1936 Dvorak Keyboard Layout by August Dvorak 

1937 World Brain by H. G. Wells 

1938 Biro Pen by Laszl6 Bird 


1940 


1940s-60s Information as a concept, through the works of Claude Shannon (information 
theory), Warren Weaver (machine translation), Alan Turing (universal computer), Norbert 
Wiener (cybernetics) and Friedrich Hayek (invisible hand is information) 

1942 Xerography Patent by Chester Carlson. The technique was originally called 
electrophotography 

1943 The term ‘acronym’ coined, meaning word formed from the first letters of a series of 
words 

1944 Marking pen which held ink in liquid form in its handle and used a felt tip by Walter J. 
De Groft which becomes ‘Sharpie’” in 1964 

1945 Memex proposed by Vannevar Bush in As We May Think 

1945 ENIAC first programmable, electronic, general-purpose digital computer by J. Presper 
Eckhart and John Mauchley (University of Pennsylvania) 

1946 A Logic Named Joe by Murray Leinster 

1946 Works on Machine Translation by Andrew Booth 

1947 Machine translation, suggested in a letter from Warren Weaver to Norbert Wiener 
1946 Electric Printing Telegraph by Alexander Bain, precursor to the fax 

1948 4 Mathematical Theory of Communication by Claude Shannon, including the word 
‘bit,’ short for binary digit, credited to John Tukey 

1948 The Human Use of Human Beings: Cybernetics and Society by Norbert Wiener. The 
word cybernetics was first used in the context of the study of self-governance of people by 
Plato and in 1834 by André-Marie Ampére to mean the sciences of government in his 


classification system of human knowledge. Here Norbert Wiener introduced the term for the 
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scientific study of control and communication in the animal and the machine 
1949 El libro mecanico by Angela Ruiz Robles 
1949 Translation memo by Warren Weaver 


1949 The Lumitype-Photon Phototypesetting by the Photon Corporation based on the 
Lumitype of Rene Higonnet and Louis Moyroud 


1949 Fr Roberto Busa starts work on computerizing his Index Thomisticus (St Thomas 


Aquinas), in the process founding Humanities computing 


1949 The Chinese Language Character Reform Association established 


1950 


ca 1950 Niklas Luhmann’s Zettelkasten system for storing and cross-referencing information 


in card indexes 
1950 Whirlwind computer at MIT including a display oscilloscope becomes operational 


1950 Computing Machinery And Intelligence by Alan Turing where he proposes the question 


‘Can machines think?’. This has become known as the Turing Test 


1950s-60s Simplified Chinese characters created by works moderated by the government of 
the People's Republic of China 


1951 Doug Engelbart’s Epiphany: “Problems are getting more complex and urgent and have 


to be dealt with collectively — we have to deal with them collectively” 

1951 Qu’est-ce que la documentation? by Suzanne Briet 

1951 Regular expressions by mathematician Stephen Cole Kleene 

1951 Linear B deciphered as a syllabic script for early Greek, by Michael Ventris 

1951 LEO I the first general-purpose business computer, Lyons Ltd, text on paper-tape 
readers and punches 

1951 UNIVAC (UNIVersal Automatic Computer) by J. Presper Eckert and John Mauchly at 
EMCC/ Remington Rand 

1952 Manchester Mark I computer Love Letter Generator by Christopher Strachey, using a 
random number algorithm by Alan Turing 

1952 Antitrust Investigations And Trial Against IBM starts, dragging on for thirty years, 
finally being dismissed in 1982. IBM will cautiously monitor its microcomputer business 
practices, fearful of a repeat of government scrutiny 

1952-4 Dot Matrix Teletypewriter developed by Fritz Karl Preikschat 


1952 ‘Love letter generator’ aimed to generate a literary text from scratch, by Christopher 
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Strachey 


1953 UNIVAC 1103 designed by Seymour Cray at the Engineering Research Associates and 
built by the Remington Rand corporation 


1953 Magic Marker by Sidney Rosenthal 
1953 The Lumitype-Photon Phototypesetting System first used to set a complete published 


book and to set a newspaper 


1954 Charactron by J. T. McNaney at Convair was a shaped electron beam cathode ray tube 
functioning both as a display device and a read-only memory storing multiple characters and 
fonts on the UNIVAC 1103 


1954 IBM 740 CRT used computers to draw vector graphics images, point by point, on 35 
mm film 1956 Keyboard and Light Pen for computer text input at MIT on the Whirlwind 
computer 

1954 The Chinese Language Character Reform Committee was founded 

1955 Teletype-setting used for newspapers 

1956 Chinese List of Simplified Characters issued by State Council 

1956 First commercial computer sold with a moving-head ‘hard disk drive’, the 305 RAMAC 
by IBM 

1956 ‘Artificial Intelligence’ (AI) term coined by John McCarthy at MIT 

1956 Sensorama, the first VR machine (patented in 1962), by Morton Heilig 


1957 COMIT string processing programming language by Victor Yngve and collaborators at 
MIT 


1957 Univers typeface family by Adrian Frutiger 


1957 The term ‘initialism’ coined, a written word formed from the first letters of other words 
in a name or phrase. NATO, where the letters are sounded as a word are regarded as 


acronyms. FBI, where the letters sound as letters, are initial-words or initialisms 

1957 Dye-Sublimation printing by Noél de Plasse at Sublistatis SA 

1957 Helvetica typeface family by Max Miedinger 

1958 The Uses Of Argument by Stephen Toulmin introduces the argumentation diagram 


1958 Lisp programming language designed by John McCarthy at MIT and developed by 
Steve Russell, Timothy P. Hart, and Mike Levin 


1958 Integrated Circuit (IC) by Jack Kilby at Texas Instruments 
ca 1958 Speed reading by Evelyn Wood 
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1960 


1960s ‘Word Processing’ term invented by IBM 
1960 Telesphere Mask which was the first head-mounted display (HMD) by Morton Heilig. 


This provided stereoscopic 3D images with wide vision and stereo sound. There was no 
motion tracking in the headset at this point. 

1960 PLATO (Programmed Logic for Automatic Teaching Operations) generalized computer- 
assisted instruction system by Donald Bitzer at the University of Illinois 


1960 Colossal Typewriter by John McCarthy and Roland Silver at Bolt, Beranek and 
Newman (BBN) 

1960 Ted Nelson’s epiphany about interactive screens becoming universal, on-line publishing 
by individuals 

1960 Suggestion for emoticon by Vladimir Nabokov 

1960 Man-Computer Symbiosis by J.C.R. Licklider at BBN 


1961 Selectric Typewriter by IBM with a ball print head instead of jamming bars, which 
could be easily replaced for different fonts and left the paper in place and moved the type ball 
instead 

1961 Information Flow in Large Communication Nets by Leonard Kleinrock 

1961 Synthesised Speech by John Larry Kelly, Jr and Louis Gerstman of Bell Labs 

1961 Expensive Typewriter by Steve Piner and L. Peter Deutsch 

1962 TECO ( Text Editor & Corrector), both a character-oriented text editor/word processor 
and a programming language, by Dan Murphy 

1962 Western Union Telegraph Company established its Telex system in the United States 
(where the name Telex is a registered trademark) 

1962 Highlighter Pen by Frank Honn 

1962 Modern fibre-tipped Pen by Yukio Horie at the Tokyo Stationery Company 

1962 Enciclopedia Mecanica by Angela Ruiz Robles 

1962 RUNOFF by Jerome H. Saltzer. Bob Morris and Doug Mcllroy (text editor with 
pagination) 

1962 The Structure of Scientific Revolutions by Thomas S. Kuhn 

1962 Spacewar! by Steve Russell in collaboration with Martin Graetz and Wayne Wiitanen 
1962 Augmenting Human Intellect: A Conceptual Framework by Doug Engelbart at SRI 
1963 Sketchpad (a.k.a. Robot Draftsman) software by Ivan Sutherland at MIT 


1963 The ‘smiley face’ by Harvey Ball, emoticon precursor 
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1963 Augmentation Research Center by Doug Engelbart at SRI 


1963 Transport font, a sans serif typeface first designed for road signs in the United Kingdom 
by Jock Kinneir and Margaret Calvert 


1963 TJ-2 (Type Justifying Program) by Peter Samson (first page layout program) 
1963 ASCII (American Standard Code for Information Interchange) a character encoding 


standard for electronic communication developed from telegraph code 

1963 ‘Hypertext’ word coined by Ted Nelson 

1963 Computer Mouse and Chorded Keyset by Doug Engelbart 

1964 ELIZA natural language-like processing computer program by Joseph Weizenbaum at 
the MIT Artificial Intelligence Laboratory 

1964 LDX (Long Distance Xerography) by Xerox Corporation, considered to be the first 
commercial fax machine 

1964 Understanding Media by Marshall McLuhan 

1964 ASCII 7-bit standard 

1964 TYPSET text formatting software used with the RUNOFF program 


1965 TV-Edit, one of the first CRT-based display editors/word processors that was widely 
used by Brian Tolliver for the DEC PDP-1 computer 


1965 Semi-Conductor based thermal printer by Jack Kilby at Texas Instruments 

1965 ‘Hypertext’ by Ted Nelson first in print, as well as first design (zipper lists) 

1965 Vision of the Ultimate Display by Ivan Sutherland presented. The concept was of a 
virtual world viewed through an HMD which replicated reality so well that the user would 
not be able to differentiate from actual reality. 

1965 MAIL Command for MIT’s CTSS, proposed by Pat Crisman, Glenda Schroeder and 
Louis Pouzin, implemented by Tom Van Vleck and Noel Morris 

1966 Object Oriented Programming by Ole-Johan Dahl and Kristen Nygard at the 
Norwegian Computing Center 

1966 Computers and the Humanities, Journal founded by Joseph Raben at Queens College in 
the City University of New York 

1966 First flight simulator for the Air Force by Thomas Furness 

1967 HES (The Hypertext Editing System) co-designed at Brown University by Ted Nelson, 
Andy van Dam and Steve Carmody, as well as other student implementors, based in part on a 
spec Ted Nelson had written previously for Harcourt Brace 

1967 The Quick-Draw Graphics System masters thesis by Jef Raskin 

1967 Logo programming language designed by Wally Feurzeig, Seymour Papert, Cynthia 
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Solomon at Bolt, Beranek and Newman 
1967 Newspapers use digital production processes and begin using computers for operations 


1968 A ‘low-tack’, reusable, pressure-sensitive adhesive accidentally created by Dr. Spencer 
Silver at 3M which would eventually be marketed as Post-it® Note 


1968 development of the ALOHA network (Additive Links On-line Hawaii Area) starts at the 
University of Hawaii under the leadership of Norman Abramson and Franklin Kuo, along 
with Thomas Gaarder, Shu Lin, Wesley Peterson and Edward ("Ned") Weldon 


1968 Doug Engelbart’s Seminal Demo of the NLS system at FJCC, including windows, 
hypertext, graphics, efficient navigation and command input, video conferencing, the 
computer mouse & chorded keyset, word processing, dynamic file linking and revision 


control 


1968 The first computer rendering virtual reality HMD, The Sword of Damocles by Ivan 
Sutherland and his student Bob Sproull 


1968 Dynabook Concept computer by Alan Kay 

1968 Digi Grotesk, digital typeface by Rudolph Hell 

1968 The Art of Computer Programming by Donald Knuth 

1968 OCR-A monospaced typeface for Optical Character Recognition by 23 American type 


foundries 


1968 OCR-B monospaced typeface by Adrian Frutiger for Monotype, following the 


European Computer Manufacturer’s Association standard 
1968 Serial Impact Dot Matrix Printer by OKI 


1968 SHRDLU natural language understanding computer program by Terry Winograd at 
MIT 


1969 FRESS, inspired in part by HES and Engelbart’s NLS by Andy van Dam and his 


students at Brown University 


1969 GML, leading to SGML by Charles Goldfarb, Edward Mosher and Raymond Lorie at 
IBM 


1969 Ed line editor/word processor for the Unix, developed in by Ken Thompson 

1969 Vladamir Nabokov presents concept of emoticon/emoji to New York Times 

1969 Structured Writing and Information Mapping by Robert E. Horn 

1969 ARPANET based on concepts developed in parallel with work by Paul Baran, Donald 


Davies, Leonard Kleinrock and Lawrence Roberts 
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1970 


1970s Gyricon Electronic Paper by Nick Sheridon at Xerox PARC 

1970 Xerox PARC founded by Jacob E. Goldman of Xerox 

1970 The Western Union Telegraph Company acquires TWX from AT&T 
1970 IBIS (issue-based information system) conceptualised by Horst Rittel 
1970 Journal by David A. Evans 


1970 Bomber by Len Deighton, first published novel written with the aid of a commercial 
word processor, the IBM’s MT/ST (IBM 72 IV) 


1970 Daisy Wheel Printing by Andrew Gabor at Diablo Data Systems allowing for 


proportional fonts 

1971 New York Times article refers to “the brave new world of Word Processing” 
1971 Laser Printer by Gary Starkweather at Xerox PARC 

1971 File Transfer Protocol (FTP) by Abhay Bhushan 

1971 Project Gutenberg by Michael S. Hart 

1971 Email with @ by Ray Tomlinson 

1971 The first ALOHA network packet broadcasting in operation 


1971 PUB scriptable markup language. Brainchild of Les Earnest of the Stanford Artificial 
Intelligence Laboratory and implemented by Larry Tesler 


1972 TLG (Thesaurus Linguae Graecae) founded by Prof Marianne McDonald at the 
University of California, Irvine, to create a comprehensive digital collection of all surviving 


Greek texts from antiquity to the present era 

1972 Computerised flight simulator by General Electric Corporation 

1972 C programming language by Dennis Ritchie and Ken Thompson 

1972 Xerox Star memo written by Butler Lampson, inspired by NLS 

1973 Xerox Alto by Xerox PARC designed primarily by Charles P. Thacker 

1973 Addison-Wesley replaces its mechanical typesetting technology with computerised 


typesetting 
1973 Copy & Paste by Larry Tessler at Xerox PARC 


1973 Click & Drag by Jeff Raskin at Xerox PARC 


1973 Micral, first personal computer using a microprocessor by André Truong Trong Thi, 
Réalisation d'Etudes Electroniques (R2E), (Orsay, France) 


1973 Community Memory Bulletin Board precursor 
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1973-1974 Ethernet as developed at Xerox PARC for Alto computers to communicate. 
Inspired by ALOHAnet, which Robert Metcalfe had studied as part of his PhD dissertation 
and was originally called the Alto Aloha Network 

1974 Omni-Font Optical Character Recognition System (OCR) Scanners by Ray Kurzweil at 
Kurzweil Computer Products 

1974 Bravo word processor by Butler Lampson, Charles Simonyi at Xerox PARC. They 
would go on to produce Word 

1974 Computer Lib/Dream Machines by Ted Nelson 

1974 ‘Writing with light, writing on glass’ were the closing words of Wilfred A. Beeching’s 
Century of the Typewriter 

1974 Transmission Control Protocol (TCP) an internet working protocol for sharing resources 
using packet switching among network nodes forming the foundation of the Internet (short 
for internet working) 

1975 ZOG by Allen Newell, George G. Robertson, Donald McCracken and Robert Akscyn at 
Carnegie Mellon University 

1975 Microsoft founded by Bill Gates and Paul Allen 

1975 MUSA Speech Synthesis systems (MUltichannel Speaking Automaton) project led by 
Giulio Modena 

1975 Ethernet patent filed by Xerox listing Metcalfe, David Boggs, Chuck Thacker, and 
Butler Lampson as inventors 

1975 The first interactive VR platform, VIDEOPLACE, by Myron Krueger 

1975 Altair 8800 computer by Ed Roberts and Forrest M. Mims III 

1975 Gypsy document preparation system/word processor by Larry Tesler, Timothy Mott, 
Butler Lampson, Charles Simonyi, with advice from Dan Swinehart and other colleagues 
1975 Colossal Cave Adventure text adventure game by Will Crowther and later expanded by 
Don Woods 

1976 Second edition of The Art of Computer Programming by Donald Knuth, published by 
Addison-Wesley, which was typeset using phototypesetting which inspired him to develop 
TeX since he found the typesetting inferior to the original, Monotype typeset edition 

1976 Frutiger series of typefaces by Adrian Frutiger 

1976 After Ethernet was deployed at PARC, Metcalfe and Boggs published a seminal paper 
on Ethernet 

1976 Apple Computer (later Apple Inc.) founded Steve Jobs, Steve Wozniak and Ronald 
Wayne 
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1976 The Metanovel: Writing Stories by Computer by James Meehan 


1976 Emacs (Editor MACroS) word processor by David A. Moon, Guy L. Steele Jr. and 
Richard M. Stallman, based on TECO 


1976 vi word processor by Bill Joy (now Vim) 
1976 PROMIS (Problem-Oriented Medical Information System) by Jan Schultz and 


Lawrence Weed the University of Vermont 
1977 Apple II computer by Steve Wozniak at Apple 
1977 DataLand developed at MIT 


1977 Zork interactive fiction computer game by Tim Anderson, Marc Blank, Bruce Daniels, 
and Dave Lebling at MIT 


1977 Inkjet Printing by Ichiro Endo at Canon 
1977 Preliminary Description of TEX Memo by Donald Knuth 


1977 Name/Finger protocol (provided status on a particular computer system or person at 


network sites) by Harrenstien 


1978 Aspen Movie Map, the first hypermedia/interactive videodisc by Andy Lippman, Bob 
Mohl and Michael Naimark of the MIT Architecture Machine Group 


1977 Personal computers as dynamic multimedia by Alan Kay and Adele Goldberg 
1978 Public dial-up BBS by Ward Christensen and Randy Suess 


1978 TeX by Donald Knuth released as the first version which was used by others. Written in 
SAIL (Stanford Artificial Intelligence Language) 


1978 American Mathematical Society Gibbs Lecture by Donald Knuth, Mathematical 
Typography; published in the Bulletin (New Series) of the American Mathematical Society, 
volume 1, 1979, pp. 337-372 


1978 Vancouver Citation Style (author number), as a part of the Uniform Requirements for 
Manuscripts Submitted to Biomedical Journals (URMs) 


1978 QuarkXPress desktop publishing software by Quark 


1978 Earliest documented electronic Spam (although the term had not yet been coined) by 
Gary Thuerk 


1978 LISA computer by Apple design starts, with a requirement for proportional fonts 
1978 Speak & Spell by Texas Instruments 

1978 Highlighters with fluorescent colours by Dennison Company 

1978 Wordstar word processor by Rob Barnaby 

1979 WordPerfect word processor by Bruce Bastian and Alan Ashton at Brigham Young 


University 
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1979 Hayes Modem by Dennis C. Hayes and Dale Heatherington 
1979 Metafont by Donald Knuth 
1979 VR integrated into pilot’s HMD, the VITAL helmet, by McDonnell-Douglas 


Corporation 

1979 -) proposed by Kevin Mackenzie as a joke-marker precursor emoticon 

1979 Architext by Genette, Gerard. Hypertext as based on a hypotekst 

1979 EasyWriter for Apple II by John Draper 

1979 TV-EDIT word processor was used by Douglas Hofstadter to write ‘Gédel, Escher, 
Bach’ 


1979 Macintosh Project started by Jef Raskin and included Brian Howard, Marc LeBrun, 
Burrell Smith, Joanna Hoffman, and Bud Tribble. Named for Raskin’s favourite apple, the 
succulent McIntosh. He changed the spelling of the name to avoid potential conflict with the 


audio equipment manufacturer named McIntosh 
1979 Post-Its® by 3M sold commercially 
1979 Steve Jobs visited Xerox PARC, organized by Jef Raskin, as part of an investment 


agreement 


1980 


1980s SPAM used as a term to describe users on BBSs and MUDs who repeat it a huge 
number of times to scroll other users’ text off the screen. It later came to be used on Usenet to 


mean excessive multiple postings 

1980s Telex usage goes into decline as fax machines grow in popularity 
1980 ZX80 by Sinclair 

1980 Stereo vision glasses by the StereoGraphics company 


1980 Smalltalk designed by Alan Kay, Dan Ingalls, Adele Goldberg and developed by Alan 
Kay, Dan Ingalls, Adele Goldberg, Ted Kaehler, Diana Merry, Scott Wallace, Peter Deutsch at 
the Learning Research Group of Xerox PARC 


1980 PC by IBM 

1980 Imagen founded by Les Earnest, sold to QMS in 1987 
1980 Floppy Disks become prevalent for personal computers 
1980 Vydec1800 Series Word Processor by Exxon 

1980 ENQUIRE proposed by Tim Berners-Lee 

1980 USENET by Tom Truscott and Jim Ellis 
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1982-3 The Encyclopaedia Project by Alan Kay, Charles Van Doren, Brenda Laurel, Steve 
Weyer and Bob Stein at Atari Research Group 


1981 Movie Manual by David Backer at the MIT Architecture Machine Group 
1981 Raskin leaves the Macintosh project and Steve Jobs takes over 


1981 BITNET, EARN and NetNorth network university IBM mainframes, allowing text 
(mail, files, chat) to be shared by non-Arpanet institutions 

1981 TPS (Technical Publishing Software) by David Boucher at Interleaf, allowed authors to 
write text and create graphics WYSIWYG 

1981 First major use of Information Murals in Organizations by David Sibbet 

1982 Guide by Peter J. Brown at Canterbury University 

1982 Adobe founded by John Warnock and Charles Geschke 

1982 First ASCII emoticons :-) and :-( by Scott Fahlman at Carnegie Mellon University 
1982 CD-ROM by Denon 

1982 First wired gloves, the Sayre gloves, by Sandin and Defanti 


1982 Tron movie released, the first movie written on a computer, an Alto at PARC. Written 
by Bonnie MacBird based on inspiration by Ted Nelson’s Computer Lib with consultation 


from Alan Kay, whom Bonnie would later marry 

1982 TeX82, a new version of TeX, rewritten from scratch, renaming the original TeX TeX78 
1983 Viewtron by AT&T and Knight Ridder 

1983 MILNET physically separated from ARPANET 

1983 ThinkTank outliner for Apple I 

1983 ARPANET switches to TCP/IP 

1983 Lisa by Ken Rothmuller, replaced by John Couch with contributions from Trip 
Hawkins, Jef Raskin and Steve Jobs, at Apple 

1983 Word word processor for DOS by Charles Simonyi and Richard Brodie for Xenix (Unix 
OS) and MS-DOS, at Microsoft. Originally called ‘Multi-Tool Word’ 

1983 KMS (Knowledge Management System), a descendant of ZOG by Don McCracken and 
Rob Akscyn at Knowledge Systems (a spinoff from the Computer Science Department of 
Carnegie Mellon University) 

1983 Hyperties by Ben Shneiderman at the University of Maryland 

1983 Multi-Tool Notepad word processor by Richard Brodie at Microsoft 

1983 ‘1984’ Macintosh Television Commercial by Apple 

1984 Literate Programming introduced by Donald Knuth, an approach to treat a program as 


literature understandable to human beings. Implemented at Stanford University as a part of 
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research on algorithms and digital typography under the name WEB 


1984 Macintosh launched. In addition to the original contributors, the team also included Bill 
Atkinson Chris Espinosa, Joanna Hoffman, George Crow, Bruce Horn, Jerry Manock, Susan 
Kare, Andy Hertzfeld, and Daniel Kottke 


1984 MacWrite word processor included with Macintosh, by Randy Wigginton, Don Breuner 
and Ed Ruder of Encore Systems for Apple. Also known as ‘Macintosh WP’ (Word 


Processor) and ‘MacAuthor’ before release 


1984 The Print Shop designed by David Balsam and programmed by Martin Kahn at 
Broderbund 


1984 Metafont by Donald Knuth updated to a version still in use at the time of writing this 
book 


1984 FidoNet bulletin board system software by Tom Jennings 
1984 LaserWriter printer by Apple 

1984 ‘Cyberspace’ term coined by William Gibson in Neuromancer 
1984 Organizer by David Potter at Psion 


1984 PostScript by John Warnock, Charles Geschke, Doug Brotz, Ed Taft and Bill Paxton at 
Adobe, influenced by Interpress, developed at Xerox PARC 


1984 MacroMind founded by Marc Canter, Jay Fenton and Mark Stephen Pierce 
1984 PC Jr desktop computer by IBM 
1984 Notecards by Randall Trigg, Frank Halasz and Thomas Moran at Xerox PARC 


1984 Highlighted Selectable Link by Ben Shneiderman and Dan Ostroff at University of 
Maryland 


1984 TIES by Ben Shneiderman at University of Maryland 
1984 LaserJet by HP 


1984 Text Messaging / SMS (short message service) developed by Franco-German GSM 
cooperation by Friedhelm Hillebrand and Bernard Ghillebaert 


1984 Filevision by Telos 

1984 LaTeX by Leslie Lamport who was writing a book and needed macros for TeX, 
resulting in ‘Lamport’s TeX’ (‘LaTeX’) 

1984 Zoomracks for Atari by Paul Heckel 

1985 Symbolics Document Examiner by Janet Walker 

1985 Guide, commercial edition, by OWL (Office Workstations Ltd) 

1985 Pagemaker desktop publishing software by Aldus, bought by Adobe in 1994 


1985 StarWriter word processor by Marco Borries at Star Division 
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1985 Intermedia by Norman Meyrowitz and others at Brown University 

1985 Windows operating system spearheaded by Bill Gates at Microsoft 

1985 Write word processor by Microsoft, included with Windows 

1985 Word word processor by Microsoft ported to Macintosh 

1985 Amiga computer by Commodore 

1985 Emacs General Public License by Richard Stallman, the first copyleft license 

1985 TRICKLE by Turgut Kalfaoglu at Ege University, Izmir; BITNET-to-Internet gateway 
allows sharing of text and programs between two disparate networks 

1986 Guide by Peter J. Brown at the University of Kent, marketed by OWL 

1986 Harvard Graphics desktop business application by Software Publishing Corporation 


1986 Texinfo GNU Documentation System by Richard Stallman and Bob Chassell, 
developed by Brian Fox and Karl Berry 


1986 FrameMaker document/word processor by Frame Technology. Developed by Charles 
‘Nick’ Corfield based on an idea from Ben Meiry and commercialised with Steve Kirsch. 
Bought by Adobe 1995 


1986 Hyperties commercial version by Cognetics Corporation 

1986 Solid Ink Printing by Tektronix 

1986 SGML (Standard Generalized Markup Language), ISO 8879 

1986 Uncle Roger by Judy Malloy released on Art Com Electronic Network on The Well 


1987 PowerPoint presentation software created by Robert Gaskins and Dennis Austin at 


Forethought Inc., bought by Microsoft same year and released as a Microsoft product 1989 
1987 MacroMind Director multimedia authoring by MacroMind 

1987 V.I.P. (Visual Interactive Programming) by Dominique Lienart at Mainstay Inc 

1987 Storyspace by Jay David Bolter & Michael Joyce, maintained and distributed by Mark 
Bernstein of Eastgate Systems 

1987 Afternoon a story, by Michael Joyce, first digital hypertext narrative 

1987 Unicode by Joe Becker from Xerox with Lee Collins and Mark Davis from Apple 
1987 Franklin Spelling Ace by Franklin Electronic Publishers 

1987 Apple Knowledge Navigator visionary concept video initiated by John Sculley, 
sponsored by Bud Colligan, written and creatively developed by Hugh Dubberly and Doris 
Mitsch with input from Mike Liebhold and advice from Alan Kay, inspired by the MIT Media 
Lab, with product design by Gavin Ivester and Adam Grosser at Apple 

1987 TEI (Text Encoding Initiative) ‘Poughkeepsie Principles’: text encoding guidelines for 


Humanities texts 
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1987 HyperCard by Bill Atkinson at Apple. Originally called ‘WildCard’. 

1987 Amanda Goodenough’s children’s point and click stories in Hypercard published by 
Voyager 

1987 Hypertext’87 First ACM conference on hypertext 

1985 VR company VPL Research, Inc founded by Jaron Lanier and Thomas Zimmerman 
1988 Microcosm by Wendy Hall, Andrew Fountain, Hugh Davis and Ian Heath 

1988 NeXT Cube by NeXT 

1988 IRC by Jarkko Oikarinen 

1988 Think’n Time (Visual outliner with dates) by Benoit Schillings & Alain Marsily at 
Mainstay Inc 

1988 # (hash) and & (ampersand) used in IRC to label groups and topics (RFC 1459) 
1988 Wolfram Mathematica by Stephen Wolfram 

1988 Hypertext edition of Communications of the ACM using Hyperties by Ben 


Shneiderman 
1988 Idex by William Nisen of Owl, based on Guide 
1988 Hypertext Hands-On! by Ben Shneiderman and Greg Kearsley, first commercial 


electronic book 


1988 Reflections on NoteCards: seven issues for the next generation of hypermedia systems 
by Frank,G. Halasz 


1988 Serial Line Internet Protocol (SLIP) by J. Romkey 
1988 Breadcrumb Trail navigation metaphor in Hypergate by Mark Bernstein 
1989 GRiDPad 1900, the first commercial tablet by GRiD Systems Corporation 


1989 Robert Winter’s CD Companion to Beethoven’s Ninth Symphony, published by 
Voyager, the first viable commercial CD-ROM 


1989 Markup (Visual document annotations with markup signs - Groupware) by Dominique 


Lienart & all at Mainstay Inc 
1989 Hypercard 2.0 by Kevin Calhoun at Apple 
1989 SuperCard by Bill Appleton at Silicon Beach Software 


1989 gIBIS by Jeff Conklin and Michael Begeman, commercialised in the 1990s as CM/1 
and QuestMap 


1989 Bidirectional Email-to-Fax Gateway hosted by UCC 
1989 Word for Windows word processor by Microsoft 
1989 Mapping Hypertext: Analysis, Linkage, and Display of Knowledge for the Next 


Generation of On-Line Text and 
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Graphics by Robert E. Horn 


1989 Information Management: A Proposal by Tim Berners-Lee at CERN. World Wide Web 
protocols published on USENET in alt.hypertext 


1990 


1990s T9 invented by Martin King and Cliff Kushler, co-founders of Tegic 

1990s Compendium by Al Selvin and Maarten Sierhuis 

1990 MarcoPolo (Visual Document Management - Groupware) by Benoit Schillings & Alain 
Marsily at Mainstay Inc 

1990 Archie, a tool for indexing FTP archives, considered to be the first Internet search 
engine, by Alan Emtage and Bill Heelan at McGill University/Concordia University in 
Montreal 

1990 Python programming language by Guido van Rossum 

1990 The SGML Handbook by Charles F. Goldfarb 

1990 Designing Hypermedia for Learning by David H. Jonassen and Heinz Mand (editors) 
in which updated conference proceedings are annotated by the authors with typed hypertext 
links in the margins connecting passages between the articles 

1991 Gopher protocol by the University of Minnesota (initial version of the protocol 
appeared in 1991, codified in 1993 as a RFC 1436) 

1991 Seven Issues: Revisited Hypertext ‘91 Closing Plenary by Frank G. Halasz at Xerox 
Corporation 

1991 World Wide Web by Tim Berners-Lee becomes the first global hypertext system 

1991 DocBook DTD by HaL Computer Systems and O’Reilly & Associates 

1991 Camelot Project started as in at Adobe, later to become PDF 

1991 PowerBook Laptops by Apple 

1991 Aquanet by Catherine C. Marshall, Frank G. Halasz, Russell A. Rogers and William C. 
Janssen Jr. 

1991 Visual Basic programming language by Microsoft 

1991 Java programming language project launched by James Gosling, Mike Sheridan and 
Patrick Naughton. Originally called Oak, then Green, and finally Java 

1991 Instant Update by ON Technology 

1991 HTML by Tim Berners-Lee, influenced by SGML guid, an in-house markup language at 
CERN 
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1991 CURIA (now CELT: Corpus of Electronic Texts) first corpus in Early Irish to be 
published on the World-Wide Web by University College Cork, Ireland 


1991 Expanded Books Project by The Voyager Company 
1991 TeachText by Apple, included with System 7 


1992 First Text Message (SMS) is sent by Neil Papworth reading: “Merry Christmas” to 
Richard Jarvis at Vodafone 


1992 Veronica a search engine system for the Gopher protocol by Steven Foster and Fred 


Barrie at the University of Nevada, Reno 


1992 Lynx internet web browser by Lou Montulli, Michael Grobe, and Charles Rezac at the 


University of Kansas 
1992 Frontier by Dave Winer at UserLand Software released on Mac 


1992 OpenDoc by Kurt Piersol and Jed Harris at Apple. First code named ‘Exemplar’, then 
‘Jedi’ and ‘Amber’ 


1992 Palm Computing founded by Jeff Hawkins 
1992 The End of Books By Robert Coover, Hypertext fiction cover story in the New York 


Times Book Review 

1992 Before Writing by Denise Schmandt-Besserat 

1992 Portable Document Format (PDF) by Adobe 

1992 BBEdit word processing software by Rich Siegel at Bare Bones Software 

1993 Mosaic web browser by Marc Andreessen and Eric Bina at NCSA massively 
popularises the web 

1993 Microsoft Word word processor celebrates its 10th anniversary with 10 million Word 
users 

1993 Encarta multimedia encyclopedia by Microsoft 

1993 Hypermedia Encyclopedias sell more copies than print encyclopedias 


1993 Newton MessagePad PDA by Steve Sakoman, Steve Capps, Larry Tesler, Michael 
Culbert, Michael Tchao and others at Apple under John Sculley 


1993 Early Blog by Rob Palmer 


1993 Open Agent Architecture (OAA) delegated agent framework by Adam Cheyer et al. at 
SRI International 


1993 Georgia typeface designed by Matthew Carter and hinted by Tom Rickner for Microsoft 


1993 Searching for the Missing Link: Discovering Implicit Structure in Spatial Hypertext by 
Catherine C. Marshall and Frank Shipman. First occurrence of Spatial Hypertext in print 


1993 AppleScript launched with System 7 by Apple 
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1994 PDF made freely available 

1994 Links.net blog by Justin Hall, before the term would be used 

1994 TrueType Open by Microsoft 

1994 Point-to-Point Protocol (PPP) enabled internet communications between two routers 
directly by W. Simpson 

1994 Netscape Navigator web browser by Jim Clark and Marc Andreessen at Netscape 
Communications Corp 

1994 Scripting News by Dave Winer 

1994 Yahoo! founded by Jerry Yang and David Filo 

1994 Amazon founded by Jeff Bezos 

1994 Semantic Web vision presented by Tim Berners-Lee at the first World Wide Web 
Conference 

1994 QR Code System by the Japanese company Denso Wave, a subsidiary of Toyota 
1994 World Wide Web Consortium founded 


1994 PageMill HTML authoring by Seneca Inc., bought by Adobe one year later, 
discontinued 2000 


1994 VIKI: Spatial Hypertext Supporting Emergent Structure by Catherine C. Marshall, 
Frank M. Shipman II, James H. Coombs 


1994 4 Subversive Proposal by Stevan Harnad at the University of Southampton 

1995 WordPad word processor by Microsoft is included in Windows 95, replacing Write 
1995 Netscape goes public and gains market value of almost $3B on first day of stock market 
trading 

1995 The World Wide Web Handbook by Peter Flynn, first comprehensive book on HTML 
1995 Ruby scripting langauge by Yukihiro ‘Matz’ Matsumoto 

1995 Windows 95 operating system by Microsoft 

1995 WikiWikiWeb, the first wiki, by Ward Cunningham 


1995 Java public release by James Gosling at Sun Microsystems (since been acquired by 


Oracle), the first programming language to use Unicode for all text 

1995 JavaScript by Brendan Eich at Netscape (orignally called Mocha, then LiveScript and 
later JavaScript) 

1995 AltaVista founded by Paul Flaherty, Louis Monier, Michael Burrows and Jeffrey Black 
1995 FutureSplash by FutureWave, sold to Macromedia in 1996 and renamed Flash 

1996 Cascading Style Sheets (CSS) by Hakon Wium Lie and Bert Bos at the World Wide 


Web Consortium 
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1996 Palm OS PDAs including the Graffiti handwriting system 

1996 Vaio laptop by Sony 

1996 Cyberdog OpenDoc based Internet suite of applications by Apple 

1996 OpenType by Microsoft joined by Adobe 

1996 Anoto by Christer Fahreeus to provide digital pen capability to paper 

1996 Hotmail email system by Sabeer Bhatia and Jack Smith, bought by Microsoft in 1997 
1996 The Internet Archive by Brewster Kahle 


1996 GoLive HTML authoring software by GoNet Communication, Inc., bought by Adobe 
1999 


1996 TextEdit word processor by Apple. Not meant for use, it was sample code 


1996 Live word count by Keith Martin, demonstrated in the Wordless word processor, later 


appearing in Microsoft Word 98 


1997 Emoji developed by Japanese mobile operators during the 1990s including SoftBank 
and Shigetaka Kurita for i-mode 


1997 Meta Content Framework developed by Ramanathan V. Guha at Apple Computer’s 
Advanced Technology Group, leading to RDF 


1997 OpenDoc by Apple cancelled 
1997 Apple Data Detectors by Jim Miller, Thomas Bonura and others at Apple’s Advanced 


Technology Group, which would also lead on to LiveDoc 


1997 Resource Description Framework (RDF) derived from W3C’s PICS, Dublin Core and 
from the Meta Content Framework (MCF) developed by Ramanathan V. Guha at Apple and 
Tim Bray at Netscape 


1997 Dreamweaver HTML authoring software by Macromedia, bought by Adobe 2005 
1997 Yandex by Arkady Volozh and Ilya Segalovich 

1997 Flash multimedia authoring and platform by Macromedia, later bought by Adobe 
1997 ‘weblog’ term coined by Jorn Barger to describe a log of his internet activity 

1997 Jabberwacky released online by Rollo Carpenter 

1997 E-Paper by Barrett Comiskey, Joseph Jacobson and JD Albert at E Ink Corporation 
1997 Newton PDA by Apple cancelled after Steve Jobs return 

1997 Unistroke by David Goldberg at Xerox PARC 


1997 90001 Communicator monile phone by Nokia, the first mobile phone with a full 
keyboard 


1997 OpenType by Microsoft 
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1997 Liquid Mail email system by Frode Alexander Hegland featuring smart Views 
1998 iMac desktop computer by Apple 

1998 First blog published on an established news site by Jonathan Dube at The Charlotte 
Observer 


1998 Can Computers Think? History and Status of the Debate. Seven posters. Industrial 


strength argumentation map by Robert E. Horn 
1998 Open Diary blogging service by Bruce Ableson 
1998 Visual Language: Global Communication for the 21st Century Robert by E. Horn 


1998 (possibly 1999) Fluid Links demo video at the ACM CHI conference by Polle T. 
Zellweger, Bay-Wei Chang, and Jock D. Mackinlay 


1998 ‘SPAM’ in The New Oxford Dictionary of English 
1998 Google founded by Larry Page and Sergey Brin 
1998 XML 1.0 becomes a W3C Recommendation 

1998 Netscape goes open source with the name Mozilla 


1998 XML-RPC text-based networking protocol between apps running across operating 


systems 

1998 Frontier blog software by Dave Winer at UserLand Software released on Windows 
1998 MathML by W3C 

1998 @font-face by W3C 

1998 AOL buys Netscape for $4 Billion 

1999 Open eBook 

1999 The short form, ‘blog’, was coined by Peter Merholz. Shortly thereafter, Evan Williams 


at Pyra Labs used ‘blog’ as both a noun and verb and devised the term ‘blogger’ in 


connection with Pyra Labs' Blogger product, leading to the popularization of the terms 
1999 LiveJournal blogging service by Brad Fitzpatrick at Danga Interactive 


1999 Blogger blogging service by Evan Williams and Meg Hourihan with significant coding 
by Paul Bausch and Matthew Haughey 


1999 RDF Site Summary (RSS 0.9) the first version of RSS, by Dan Libby and Ramanathan 
V. Guha at Netscape 


1999 RSS 0.91 by Dave Winer at UserLand 

1999 my.netscape.com and my.userland.com 

1999 Edit This Page by Dave Winer 

1999 Code and Other Laws of Cyberspace by Larry Lessig 
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1999 Mac OS X operating system by Apple 
1999 Ajax web development techniques for asynchronous web applications emerges 


1999 ActiveText: A Method for Creating Dynamic and Interactive Texts by Jason E. Lewis 


and Alex Weyers at Interval Research Corporation 


1999 Spatial Hypertext: An Alternative to Navigational and Semantic Links by Frank M. 
Shipman and Catherine C. Marshall 


1999 Electronic Literature Organization (ELO) founded by Scott Rettberg, Robert Coover, 
and Jeff Ballowe 


2000 


2000 Optical Character Recognition (OCR) software is made available online for free 

2000 1 billion indexable pages on the Web, estimated by NEC-RI and Inktomi 

2000 ClearType by Microsoft 

2000 XML Linking Language (XLink) an XML markup language for creating internal and 
external links within XML documents, and associating metadata with those links, by Steven 
DeRose, Eve Maler, David Orchard and Bernard Trafford 


2000 EPrints by Stevan Harnad, funded by Wendy Hall, supervised by Les Carr and 
implemented by Rob Tansley and others at the University of Southampton 


2000 CoolType by Adobe 


2000 ScholOnto by Simon Buckingham Shum, Enrico Motta and John Domingue at the 
Knowledge Media Institute, The Open University. This evolved over the next decade into 
ClaiMaker and Cohere with Victoria Uren, Gangmin Li, Anna De Liddo and Michelle 
Bachler 

2000 Riding the Bullet by Stephen King, the first mass-market e-book for encrypted 
download 

2000 EverNote founded by Stepan Pachikov 

2001 ‘Chinese General Language and Character Law’ rolled out. 

2001 Tinderbox by Mark Bernstein, Eastgate Systems 


2001 Semantic Web vision popularised in a Scientific American article by Tim Berners-Lee, 


James Hendler and Ora Lassila 
2001 G4 Titantum PowerBook laptop computer by Apple 
2001 The Wiki Way by Bo Leuf and Ward Cunningham 
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2001 Creative Commons by Lawrence Lessig, Hal Abelson, and Eric Eldred 

2001 Wikipedia online collaborative encyclopedia by Jimmy Wales and Larry Sanger at 
Nupedia 

2001 Movable Type weblog publishing system by Benjamin Trott and Mena Grabowski Trott 
at Six Apart 

2001 JSON by Douglas Crockford 

2001 Douglas Adams’ speech about Virtual Graffiti held at the 3GSM World Congress 

2002 Bibliotheca Alexandrina founded, the modern Library of Alexandria, with Ismail 


Serageldin as the founding director 

2002 EPrints version 2 lead developer Christopher Gutteridge 

2003 Android Inc founded by Andy Rubin, Rich Miner, Nick Sears, and Chris White 
2003 Friendster social media service Jonathan Abrams 


2003 Myspace blogging and social media service by Brad Greenspan, Josh Berman and Tom 


Anderson at eUniverse 
2003 Deep Love by Yoshi, first cell phone novel ( Japanese ‘Keitai Shousetsu’) 
2003 The Legal Deposit Libraries Act widens the definition of what publishers should send to 


the libraries to include digital publications, pending further regulation 

2003 WordPress blogging service by Matt Mullenweg and Mike Little 

2003 Blogger blogging service is bought by Google 

2003 TypePad blogging service by BizLand, later Endurance International Group (EIG) 
2003 Ulysses word processor by Max Seelemann and Marcus Fehn 


2004 Facebook social media service by Mark Zuckerberg, Eduardo Saverin, Andrew 
McCollum, Dustin Moskovitz and Chris Hughes 


2004 First hypertext format full length articles accepted at ACM’s Hypertext Conference with 


Twin media: hypertext structure under pressure by David Kolb awarded ‘Best Paper’ 


2004 First hypertext format article at ACM’s Document Engineering conference by James 


Blustein and Mona Noor 

2004 Institute for the Future of the Book founded by Bob Stein 
2004 Tag Cloud at Flickr, Technorati, WordPress Plugins and more 
2004 Scala programming language by Martin Odersky 

2005 Pages word processor by Apple 

2005 Markdown by John Gruber collaboration with Aaron Swartz 
2006 Time Person of the Year is ‘You’ 
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2005 Writely by programmers Sam Schillace, Steve Newman and Claudia Carpenter at 
Upstartle 


2006 Upstartle bought by Google 
2006 Google Docs by Google 


2006 Twitter social media service founded by Jack Dorsey, Noah Glass, Biz Stone and Evan 


Williams at Twitter 
2006 One Laptop Per Child by Nicholas Negroponte 


2006 HyperScope Project by Doug Engelbart and Brad Neuberg, Eugene Kim, Jonathan 
Cheyer and Christina Engelbart 


2006 Hyperwords Project by Frode Hegland, Fleur Klijnsma and Rob Smith 
2006 Office Open XML by Microsoft 


2006 The Semantic Web Revisited by Tim Berners-Lee, Nigel Shadbolt, and Wendy Hall, in 
IEEE Intelligent Systems 


2006 Debategraph by Peter Baldwin and David Price 


2006 Gamer Theory by McKenzie Wark’s, the first networked book, produced by the 
Institute for the Future of the Book 


2006 Dialogue Mapping: Creating Shared Understanding of Wicked Problems by Jeff 
Conklin 


2007 Hashtag by Chris Messina (name by Stowe Boyd) 

2007 iPhone by Apple Inc. 

2007 Kindle by Amazon 

2007 Scrivener for macOS by Keith Blount at Literature & Latte 
2007 EPUB by IDPF 

2007 Google Street View 

2008 MacBook Air by Apple 

2008 Last Stable Build of Netscape Navigator 

2009 Like Button by Facebook 

2009 Webfonts by Typekit 

2009 OmmWriter by Herraiz Soto & Co 

2009 iPhone Copy & Paste by Apple 

2009 Twine open-source tool for authoring interactive fiction by created by Chris Klimas 


2009 Worst year in decades as far as advertising revenues for newspapers and newspapers 


begin moving online 
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2010 


2010 Thumbs Up Emoji 

2010 Retina Display by Apple 

2010 1A Writer word processor by Oliver Reichenstein 

2010 iPad tablet by Apple 

2010 Swift programming language development by Chris Lattner, with the eventual 


collaboration of many other programmers at Apple 

2010 Stereoscopic 3D mode for Google Street View 

2010 Siri developed by Dag Kittlaus, Tom Gruber, and Adam Cheyer, bought by Apple 
2010 Emoji ratified as part of Unicode 6.0 

2011 iMessage by Apple 

2011 ByWord word processor by Metaclassy 

2011 Scrivener word processor for Windows by Keith Blount at Literature & Latte 
2011 Annual Future Of Text Symposium by Frode Alexander Hegland launched 

2011 Liquid text utility by Frode Alexander Hegland at The Liquid Information Company 
2011 Siri personal digital assistant released as part of the iPhone 4S by Apple 

2011 Swype by Cliff Kushler allows users to drag their fingers on a virtual keyboard to 


connect the dots between letters 


2011 ClaiMaker by Gangmin Li, Victoria Uren, Enrico Motta, Simon Buckingham Shum and 
John Domingue 

2012 Knowledge Graph by Emily Moxley, Google’s lead product manager, at Google 

2012 Muse by Adobe 

2012 The Web-Extended Mind by Paul Smart 

2012 Palmer Luckey launched a Kickstarter campaign for the Oculus Rift which raised $2.4 
million 

2012 Inventing on Principle presentation by Bret Victor 

2012 Google Now Assistant launched by Google 

2012 Medium online social publishing platform by Evan Williams 

2012 LiquidText by Craig Tashman 

2012 Outlook by Microsoft replaces Hotmail 

2013 Non-Print Legal Deposit Regulations further define the digital elements of the Legal 


Deposit Libraries Act and lead to large-scale on-going transfer of e-journals and e-books to 
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the legal deposit libraries for posterity 
2013 Distant Reading by Franco Moretti 


2013 First Full-Scale Harvest of the UK Domain by the UK Web Archive, using the Non- 
Print Legal Deposit Regulations 


2013 Ulysses III (major rewrite) by Max Seelemann and Marcus Fehn 
2014 Xanadu by Ted Nelson 

2014 Facebook bought the Oculus VR company for $2 billion 

2014 Alexa assistant released by Amazon 

2014 Cortana assistant released by Microsoft 


2014 Framtidsbiblioteket (The Future Library project) launched, a public artwork that aims to 


collect an original work by a popular writer every year from 2014 to 2114 


2014 Author reboot by Frode Alexander Hegland at The Liquid Information Company with 
coding by Jacob Hazelgrove 


2014 Most up to date version of TeX is 3.14159265 as of the publication of this book 

2014 Swift programming language launched at the Apple Worldwide Developers Conference 
(WWDC) 

2014 Author early release by Frode Alexander Hegland at The Liquid Information Company 
2014 Augmented Writing by Textio 

2015 Notion by Ivan Zhao at Notion Labs 

2015 Watch by Apple 

2015 Hamilton musical, by Lin-Manuel Miranda, makes it Broadway debut, highlighting the 


beauty and power of the written word, with an opening line stating that Hamilton “put a 


pencil to his temple, connected it to his brain” 

2015 OpenAI founded 

2016 Reactions, also-called Tapback, for iMessage by Apple 

2016 Universal Clipboard by Apple 

2016 Viv Labs, developed by Dag Kittlaus, Adam Cheyer and Chris Brigham, acquired by 


Samsung 

2016 Notion founded by Ivan Zhao and Simon Last 

2017 Roam Research founded by Conor White-Sullivan 

2017 Web Annotations Standardised by the W3C Web Annotation Working Group 
2018 Bixby Marketplace, an open assistant ecosystem based on Viv Labs Technology, 


launched by Samsung 


364 


2018 Foundation models, which are large language models (LLM) trained on vast quantities 
of unlabeled data that can be adapted to a wide range of downstream tasks, began to be 


developed 
2019 Reader PDF viewer with Visual-Meta support by Frode Alexander Hegland at The 


Liquid Information Company with coding by Jacob Hazelgrove 
2019 GPT-2 released by OpenAI 


2020 


2020 Muse by Adobe discontinued 

2020 Flash by Adobe discontinued 

2020 iPad Keyboard with Trackpad by Apple 

2020 Adobe Liquid Mode for Easier PDF Viewing on Mobile Devices powered by Sensei 


Machine Learning 

2020 GPT-3 released by OpenAI 

2022 Gato released by DeepMind 

2023 (Jan 1), Adobe Type 1 (Postscript) fonts reach end of life; no further support in Adobe 


products (other software unaffected) 
2023 GPT-4 released by OpenAI 
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Future 


unknown The “absolutely final change (to be made after my death)” of TeX will be to change 
the version number to 2, at which point all remaining bugs will become features. Likewise, 
versions of Metafont after 2.0 asymptotically approach e (currently at 2.7182818), and a 
similar change will be applied after Knuth’s death. 

unknown All the pioneers of digital text will die, leaving it to future generations to rediscover 
and hopefully improve upon how we interact with our textual knowledge, and each other. 
unknown You will read this. What will you do with what you have learnt in this book, what 
will you think of the way we saw text in 2020, how do you think the way we present and 


interact with text can be improved? 


Contributors 


Frode Alexander Hegland, editor, with Mark Anderson, Peter Flynn, Mark Bernstein, Bernard 


Vatant, Robert E. Horn, Jonathan Finn, Niels Ole Finnemann 
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Appendix : Reading in Reader 


Please remember that you can 6-[ to return where you clicked on the link to get to this page. 


Citing 
¢ 36-C on selected text to Copy as plain text 
¢ 36-shift-C on selected text to Copy the text with full citation information for pasting in 


Author or other software. You can specify in Settings if this should be in the BibTeX 


format 


Annotation 

¢ Select text and ‘H’ to highlight text in grey 

¢ Select text and cmd-shift-H to highlight text in yellow 

¢ ‘H’ without selecting text to see all the highlights in the document or fold the document 
¢ Select text and ‘X’ to add a strikethrough to the text 

¢ When you fold the document to see headings, all the highlighted text will also appear 

¢ 36-double-click to add an annotation. This results in a clickable yellow box with your 


comments. We hope to improve on this, to at least let the first few words of what you 


wrote be visible even in this little yellow box. 


Interacting with Citations 

Some of the citations in this volume were added in Author and are thus live. For example, 
citing the Future of Text Volume 1| [49] you can click on the number in the hard brackets and 
expect to see information about this book. If you click on the title, you will be taken to the 
download page, but if you already have this document in your Reader Library it will open 


your own copy. 


A note on the design: Attractors & Distractors 


One thing I have noticed is that there there are not only visual attractors, such as bold text in, 
text of high colour contrast etc., there are also visual repellers — or ‘distractors’, such as 


banner ads on websites which we become accustomed to actively ignore, at a small, but not 
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zero, cognitive effort. 


This is why we removed the title of the document when you fold this book into a table 
of contents in Reader. Since you already know the title of the document you are reading when 
you fold it, the title and other extraneous information will act as distractors which will repel 
your eyes. We have further added a blank space above and below the table of contents, to let 
you know without having to actively think about it, whether you are at the beginning, middle 


or end of the document. 


Keywords in Colour 


On #-shift-K you will see keywords in colour, under each heading when they appear, as an 
attempt to give you a glanceable view of what that section contains. 

The idea is not that the colours are going to necessarily be very readable in this view, 
and it would be a little presumptuous of us to think you will want to take the time to learn 
what each colour corresponds to: which is why the colours are only colours and not listed by 
labels. The goal of the colours is simply that over time, when you see them in this folded 
overview, you will get more of a sense of what each colour represents, not as logical one-to- 


one representation necessarily, but as a more of a feeling. 


For example, this work on the future of text currently includes more discussions on VR 
and AI than before, and in this view the text which refers to VR will be in red whereas AI will 
be in dark green. 

There are very few colours that even those of us who do not have a form of colour 
blindness can distinguish when the colours are not next to each other; and there are very few 
colours which are highly legible on light and/or dark backgrounds. But the aim here is not for 
full legibility nor for one solution to fit all, but rather an experiment for how we can make 
text more visible for overviews without having to ‘literally’ read the text. A point could be 
made for not showing the keywords at all, only dots or lines, but we have taken this approach 
with coloured words since this will, perhaps, over time help you learn what the different 
colours denote. A further note on the colours is that we have chosen for many to be in pairs, 
such as light blue and dark blue and light green and dark green, but also red and light red 
(pink) and orange and dark orange (brown), in order to group categories together a little bit. 
For example, we are experimenting with dark blue for the names of those who have authored 
an article for either a book or Journal issue, and light blue for those members of the 
community who have provided serious input and dialog but have so far chosen not to write an 
article. For similar reasons, architectural terms are in orange since that relates to the red of 
virtual architecture and knowledge work tools and techniques (building mental structures) in 


dark orange/brown. 
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Note: You will get a different colour keyword display on doing option-shift-K as one of 
the experiments, something we may have changed yet again when you are reading this since 
we are definitely not done with this... 

At the time of writing this, | am waiting for a new version of Reader to test a new way 
to look through a document (now available to download). The issue is that text is great for 
linear reading and also great in a graph or map view, but not very good for representing 
overviews”. For something like our books and journal, we need a more dynamic and non- 
linear way for the user to be able to get clued in to what different articles are about. 

In Reader you can already perform ‘Find’ commands which results in only showing the 
lines of text with the search term, as well as keyboard shortcuts to show only lines with 
names or lines with glossary terms. This is useful but still not very ‘glanceable’ and the text 
requires reading to see what’s where. Another interaction which is currently possible and 
useful is that with PDF documents which have Visual-Meta there is also a different way to 
thumb through the pages, letting you use right arrow key for next page but down arrow key to 
go to the next article, which has turned out to be very useful in going through articles at pace, 
without having to spend mental effort skipping pages to go to the next article. But anyway, 


back to the colourful point. It is a simple affordance: 


¢ The user has a keyword colour-dictionary (initially defined by the producer of the 
software—me—but fully editable by the user, as shown opposite) where sets of simple 
keywords are grouped to have specific colours. For example, anything relating to VR is 
red while anything related to AI is dark green. Knowledge work tools and techniques are 
in dark orange/brown and architecture terms in orange. Yellow is left blank for the user to 
assign to important keywords of their choosing. 

¢ When the user performs cmd-shift-K the document will show all headings with colour 
keywords under each heading or the colour keywords will list any heading beneath them, 
to be experimented with. If you are reading this in Reader, please try it now and you’ll see 
what we ended up with! 


We initially experimented with this in the folded view but it became cumbersome. 


The possible set of colours is larger than a person can reasonably distinguish without 
seeing the colours side by side (dark blue and light blue come to mind). Some of the colours 
are not easy to read on a white background (yellow and pink for example). But this is slightly 
beside the point. 


The expectation is that a user could choose to focus on one or two colours to start with, 
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such as red and green, as mentioned above (red for VR and green for AI). I don’t think it will 
take long for that user to start to associate the colours with the themes so that when they fold 
into the outline they can get the gist of what articles primarily cover what content. After all, 
an article discussing architecture in VR would show a series of red and orange keywords, 
while an article discussing AI and knowledge work would have a series of keywords under 
the heading in dark green and brown. 

Doug Engelbart told me to try colour coding in this way many, many years ago and I 
could never make it pleasant, since reading in colour is distracting to me, but basic tests 
showed the concept made sense. What I think I have done here is providing a way to have 
a useful colour-based overview while not at all intruding on the reading experience. 

The next step, if this turns out to make sense, is to try further views, or ‘ViewSpec’s as 
Doug used to call it, to help the reader navigate and browse large compendium documents, 
primarily. I am a self funded developer with a very small budget and that may have been a 
blessing in this case since I needed to use as much of the components we already have to 
build this, and that helps the user as well, since the interaction becomes more integrated into 
the current work practices in Reader. It will be interesting to see. 

So I am writing this the day before Apple introduces their Headset, sitting in the bottom 
of the garden on a sunny Sunday, typing away on my (relative to VR) tiny 13” screen. I don’t 
mind this screen for typing, but the 27” Studio Display in my office is a different world when 
editing. The dream, for me, is to be able to work on either, and when I really need to think, 
put on a headset and fade everything but my computer screen, which I might grab and pull up 
to be a bit taller, and write. VR should, initially, be a powerful extension of the work 
environment we already use, and will likely continue to use. But when I put on a headset to 
think and concentrate, I want to be able to use the full environment, including such 
colour coding as mentioned here, to have a much more powerful, glanceable environment 
outside the ‘main display’, with what I am working on and my resources rendered in useful 
forms in the background, to the sides and so on. 

The power of VR will be to escape information flatland when desired, but it will be key 
to work in any mode I want. This is why I developed Visual-Meta, though the colours should 
be for the reader to decide. 

Tomorrow I'll see Apple’s headset, Tuesday I expect to be able to test the keyword 
colour-dictionary and Wednesday there is an upgrade to my favourite PlayStation game, 
Battlefield, where an immense explosion of interactivity is possible, where I can run and gun 
and fly and drive and be a minor superhero. Will we be able to make working in VR as 
exciting, will we be able to make working in VR even more richly interactive? I most 


sincerely hope so, but for that to happen we will need to dream, experiment to experience and 
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keep questioning, which is exactly what the Future Text Lab community will continue to do. 


Join us, if you are not already aboard: https://futuretextlab.info 
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Appendix : Symposium 2023 Transcripts in Full 


AI transcripts from the 2023 Symposium on the Future of Text from sonix.ai with only light 


editing: 
Transcript: Frode Hegland Intro 


I want to talk a little bit about us as a kind of a community. So I want to talk about the tone of 
the community. It feels weird sitting down, but with this, I kind of have to. So yesterday, my 
six year old son read this poem in school. Now we are six by AA Milne. So of course as 
parents we wanted to research this. Aa Milne, you know, very famous guy, but we looked him 
up and middle name, same as mine. That was fun. But more interesting, his teacher was H.G. 


Wells. That was kind of cool. But guess who he played cricket with? 


Frode Hegland: Okay. So that was kind of a cool surprise. You probably know at least half of 
them. So that made me think of our community. And it made made me think of lunatics 
because there was a lunar society of Birmingham in the 1700s, late 1700s. And just have a 
look at the names. That's Charles grandfather, I think. So when I look at the community, what 
we're trying to do is be social and nice, but also have interesting connections with a really 
diverse group of people, which is becoming more and more important. So I was also 
thinking, what would it be like to be at Doug's lab at Sri in the 60s? You know, to sit there 
over coffee and have those discussions or at Xerox Parc? Livia can tell us some of that. She 
was there. What was it like to be at Brandel five? Okay. I need to know how many people 
here know what Bandley five was. Really? 


Frode Hegland: That's where they developed the Macintosh. That was where the Pirates of 
Bandley 5. I'm embarrassed and in the best possible way. Anyway, So to put it in other words, 
more explicitly what we're trying for distributed because obviously we're not all together all 
the time, casual, friendly and passionate about the future of thought, primarily with text and 
extended in many directions. Today is a lot about XR, VR and all of that good stuff, but by no 
means by itself. That's really, really important to reemphasize. The reason for the XR is partly 
to cash in on the hype, to be honest, because not a lot of people care about text. So now we 
have a hype thing that people either passionately agree or disagree with, which is fine. At 


least they have an interest in it, right? So that's me saying thank you for being here. I'm 
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extremely grateful. That includes you online guys as well. Of course. And vent. Actually, I do 
have a few more words, but you'll talk after me, right? Yes. Yeah. So when we're talking 
about the future of text and XR, it is really important to remind ourselves that text already 
extends our reality. And as I just said, bra is trendy so it can help us dream again. And the 
dreaming thing is so important. The ones who've been spending the most time with me 
recently have heard me go on and on because we need to dream again for text and an 
extended space. You know, Doug and Ted and all these people in the 60s and 70s did amazing 
stuff and then we ended up with this. You know, applications on a laptop, that's we could 


have done more. So we need to dream again for traditional space as well. 


Frode Hegland: There's so much that can be done in traditional displays. We should not leave 
that behind. When we talk about VR, we're talking about immersive environments. We really 
don't care how they become immersive. We could be projectors, implants, glasses, headsets, 
whatever. And we're primarily talking about interactions with knowledge. I like to think of 
the headsets as thinking caps. We're not going to be wearing them all day in the future. But 
just like sometimes all we need is our watch. Sometimes we'll need a full environment to 
think and we take it off. So here's the thing. We need to be careful with what we build in 
terms of tools. Of course, infrastructures, which a lot of us have talked about, and maybe 
most importantly, people's expectations. Danilo What colors have I got on the screen? 
Ukraine colors. I put that I didn't know if you're going to be here today. But I think it's 
important, even when we're sitting here talking about information, to acknowledge the 
realities of the day. There are real problems we need to deal with. So Slava Ukraina. Text and 
VR? Well, yes. Um. Yulia, who might be online or not, joined us before, and this is one of his 
pieces. It's absolutely crazy. It looks horrible here because of the low quality video. It's really 
hard to capture the experience. As many of you know. But one of the amazing things is not 
just to see nodes connected. If you make a fist, that means you grab the whole world. So if 
you move it like this, the whole entire workspace moves towards you. I think that's mind 


boggling. It just makes it literally graspable. 


Frode Hegland: So we are now at the point where this stuff isn't theory. People use it and 
quite a few of us today have headsets are the ones who haven't been in it. Later today will be. 
Putting them on. And what I hope we do today and tomorrow and in general is to come up 
with better questions together. The together thing is really important. Doug Engelbart talked 
about augmenting human intellect so we could solve urgent, complex problems. Collectively. 
And the more I look at our community, the more I see we need diversity of thoughts from 


many different perspectives. And somebody mentioned once something about, Oh, we can't 
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just let anyone in. It was meant in a kind way, and we realized we can't judge that because we 
don't know what the criteria is for someone else to join the community. And somehow they're 
passionate about text and thinking. If we disagree with everything they say, all the better, 
right? So this is a call to arms to think about thoughts in multiple dimensions of freedom. 
And this is the thing. This is what's so incredible right now. This is a magic space. It's mostly 
in the imagination. The imagination is being shriveled by Apple and Meadow because they're 
defining it. Everything they define an interaction. We have less opportunity for thinking. It's 
becoming more and more of a commercial space. Do you remember the early web? You 
know, this is the freedom, space and blah, blah, blah. And what do we have today, You know? 
That's not dwell on that. So we must continue to nourish the most powerful human capability. 


Just to check if you're not asleep already. What do you think that is? Close. 


Frode Hegland: Well, that's the story. So my, my my early mentor was Doug Engelbart. And 
he once complained that someone said, Doug, you're just a dreamer. And he got really angry. 
And he said, Dreaming is hard work. And I've always thought that was a nice quote and it 
was nice. But now that we're looking at XR for a year and a half now in this community, 
we've been really trying to build better things and we've achieved almost nothing. Dreaming 
really, really is hard just to say that this entire room can be knowledge. Once you get down to 
it. What does that mean? It's really hard. So I do ask you what can happen when the 
constraints of the substrate disappears, right? Because we cannot easily separate environment 
from tool anymore. I have a thing over whether I'll show you later or information from 
interaction. So let's just look at a few specific questions. A document in VR, a traditional 
document, open it. Comfortable reading, distance floating in space. That's nice. So then I do a 
thing and all the pictures go there and the graph goes there. That's nice. Something like this. 
This is what we've been experimenting with. So then we get questions of geometric or 
semantic addressability. What is that picture on the wall? Is that defined as the picture wall? 
Is it the north wall? Is it just that rectangle that's going to have huge repercussions for how 
we go from space to space? And of course, these are all generative AI, in case you're 


wondering. 


Frode Hegland: But how cool is this one, right? What might symbol manipulation some 
people call text symbol manipulation at scale? What will it be? That's the big question for us. 
And what about links and connections? What will it mean to link? What will it mean to 
connect when it's not a thing, when it's an entire space? And then questions of tools, including 
portability and physicality in Rome for hypertext. I passed around an obsidian. It's there. I'll 
send it off too, so don't disrupt. But there's all this thing about just using hands in VR. Why in 
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the world would you want to do that? I think that's entirely stupid for some interactions. Yes, 
it's nice to look at something and do this. But seriously, could you imagine any tool person in 
the history of our species not wanting to hold something right? So we need to hold different 
types of tools, not just games controllers. So that's going to be interesting. Also, if you have a 
tool in one environment and you really like it, can you take that tool somewhere else? You 
can take a pen from Photoshop somewhere else today, but can you and should you be able to 
do that in this environment? So then we get questions of transfer and access and shared 


spaces. 


Frode Hegland: If we're in a VR environment. Now, what are you seeing? Are you seeing this 
as data or are you seeing this as an image? Is some of it blurred out because you don't have 
the access? These are important questions. And then Mark, we're walking in Rome. Asked 
what will work be in this environment? Work may become something entirely different. 
We're probably going to do the editing and the normal things we're doing now in VR, but 
work will probably change. Like work today is not the same as it was a hundred years ago. 
Similarly, what will thinking be? How will thought change when everything we see can be 
interactive? And then what will our bodies be? We will move differently because our 
movements will mean different things in their. And as a consequence, will our minds become. 
So these are some of the questions. And I think it comes down to the big O ownership, right? 
I was a teacher a few years ago with Keith over there about ten years ago, and we had a 
future of Texas at L.c.c. And I was telling the story of the early days of the Internet. And why 
bit let Vint tell it. Where are you? Where you? The big companies. Or were you when you 
developed the Internet? 


Vint Cerf: Just the point of the story. Wouldn't be companies telling you what to do. Oh, the 
companies didn't tell us what to do at all. Nobody told us what to do. That was the best 
lesson. Yeah. Investigators didn't tell us what to do. We said we went off and did something. 
And nobody stopped you. So here again, it be great for you, which is not the case for you 
right now, because in order to achieve your objectives, you have to deal with these entities 
called aliens. I wasn't supposed to be. 


Vint Cerf: Well, so specifically, what Frode is trying to do is convince people and what he is 
built is worth investing in and distributing. So there's tulan. And that turns out to require a 


commercial audience, which we didn't have to make. So your world is. Endorsements. 


Frode Hegland: In some ways, yes. Thank you. But also, you know, bringing an event like 
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this for two reasons. One is the earlier bit about our community. I mean, of all the people in 
this room, in a thousand years, your name will be remembered. You know, I'm not trying to 
butter you up, but I finally found out the origins of the phrase. But that's something else. The 
point of the story of what you and Bob Kahn did created the underlying system for 
interconnected networks. That means that all of this is possible. And the other reason I 
brought you in on this with ownership, you know, with the community we have venture, for 
crying out loud, you know, we have the ability, if we show the passion to reach out to 
different kinds of people. That is really, really important. And then finally. Who were they at 
the time? Nobodies. Right. That is really, really important. So the work is what matters. So 
and kind of closing VR extends our potential and that is Keith holding up the headset for my 
son. So how we choose to extend ourselves defines who we are and who we want to be. 
That's really, really, really important. We're living in an age of weird politics. But with this 
stuff, we can choose us and everybody who was involved with this to extend and choose who 
to be. It's protean. It's amazing. It's fantastic. And that goes to the dreaming. We have to 
remind ourselves of that. So apparently I have a final point. The most fundamental aspect of 


the universe. Anyone? Anyone? 


Vint Cerf: Ferris Bueller existence, for one thing. 


Frode Hegland: Oh, that's nice. Oh, for crying out loud. You to go somewhere else. The most 
fundamental aspect of existence. It does not matter. Or information. But interaction. Without 
interaction. There is nothing else. And we're building and talking about building fully 
interactive environments. So let's use this most fundamental aspect of what is to push it as far 
as we can. So welcome, everybody, and thank you for being here. And now we need to find 


out who would like to present first. 


Speaker3: It's more. The sound was good. It's good from you. But if you. If it was supposed 
to have been improved by passing Vint the device, guess maybe your your iPhone didn't seem 


to change it. 


Frode Hegland: The Yeah, the vent thing was passed with this, so that's an issue. The big Yeti 
microphone is for here, so we have to be careful with that. Sorry about that. Yes. On and 
morning Brandel. It's middle of the night for you, isn't it? Anybody online want to present 


now or anyone here want to go first? Livia, you want to go? For anyone else. Ready? 
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Speaker3: I don't think can go whenever you like. 


Frode Hegland: Oh, Are you going? Okay. Right. I will just hand it on to Dave because he's 


sitting right next to me. Are you? How are we communicating? Point on that piece of Lego. 


Speaker3: Great building blocks. 


Frode Hegland: Building blocks? Yes. Excuse me. You know what to do. 


Vint Cerf: If you bring it up. 


Frode Hegland: Yeah. 


Vint Cerf: That's a good idea. You're presenting the next to the next. 


Frode Hegland: Yeah. 


Speaker3: Exactly. 


Vint Cerf: Only. We will be making abominable presentation. Boop. 


Speaker3: The Excellency. Go here. I've forgotten. You're not a. 


Speaker4: Powerpoint man, Frode. You've got a share here. It's airplay, right? That's great. I 


guess we have to. 


Vint Cerf: So much for plausible. I'm going to take your seat. 


Speaker4: You already. You're already sharing screen, aren't you? 


Frode Hegland: Yes. Screen share. It's all you need to do is. 


Speaker4: Okay. So grab a seat here so hopefully everyone can hear me online and can see 
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the slides. So there's talking about interactions and relationships. It just occurred to me that 


I'm. 
Transcript: Dave Millard and Mark Anderson 


Dave Millard: So my name is Dave Millard. I'm a professor of computer science at 


Southampton. 


Dave Millard: So I'm one of Frode supervisors at at the university. And I noticed that Dave 
De Roure in the corner was one of my internal examiners for my PhD. So that makes him sort 


of grandfatherly figure for me. 


Dave Millard: There you go. These connections. Connections. Okay, so apologies. A few of 
you saw a longer version of this presentation at Hypertext, but I thought it was relevant here. 
It was a this is a paper that Mark and I wrote to try and address this question of what is 
hypertext and what is it in today's world? And our motivation was that we noticed that a lot of 
people think about hypertext as a historical technology, right? So something that, you know, 
is perhaps in the 80s and 90s and we've moved on and now we're looking at other things and 
in the ACM Hypertext conference this question of what is hypertext anyway and what is its 
relevance has come up time and time again. So in 99, Mark Bernstein asked, you know, 
where are the hypertexts? 2003 Pete Nurnberg did a keynote called What is Hypertext? And 
Noah Wardrip-fruin in 2004 presented a paper which was what Hypertext is. Well, he was 
complaining about what Pete said. You know, what is hypertext? The general consensus 1s, is 
it's something to do with with links and linking. But Mark and I took a different approach. 
We we looked back through the literature and we try to identify traditions of talking about 
hypertext and try to understand what were the commonalities and what the the themes were 
and what might link these different things together. 


Dave Millard: We came up with seven different contexts and I'll let Mark touch on those later. 
But I wanted to start off with the punch line, which I think is kind of most relevant to this 
community, and I'm going to talk about that through a little story about Jean-Paul Sartre and 
SharePoint, if you'll indulge me for a moment. So for those of you who don't know, Jean-Paul 
Sartre was a French playwright and philosopher, very much associated with the philosophy of 
existentialism and existentialism. Oh, my, my my slides have broken slightly on your in your 


performance. But existentialism is essentially associated with this idea of existence precedes 
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Essence. In other words, we're not born into a world where we have a purpose and we have a 
meaning, and there is a set of rules we must follow, but rather we must build those things in 
the world ourselves, right? Very often it's associated with a sort of more atheistic position, but 
that isn't necessarily the case if you have an ideology, if you have a set of rules that you're 
following, The point of existentialism is that you should critically engage with those and 


think about what those rules mean in your modern life and your modern context. 


Dave Millard: And people like Jean-Paul Sartre were therefore fascinated by the idea of what 
does it mean to live a good life if there's no if there's no predefined purpose, if we must build 
it ourselves, what does that mean? And the answer to them was exactly this point about 
critical engagement. So they call this being authentic. So essentially, if you were engaged in 
this process of constructing this meaning and thinking critically, you were leading an 
authentic life. Whereas on the other hand, if you were kind of drifting through life without 
engaging in that thought, then you were living in bad faith. And that brings me back to 
SharePoint. So as I said, I'm at the University of Southampton. Southampton uses SharePoint 
for its information infrastructure, and I'm not a big fan of SharePoint. I don't like the way that 
it makes me structure files. I don't like the fact that I can't link between things easily or tag 
things easily. I can't share things in the way I particularly want to share them. Despite the 
name SharePoint for me in my digital life is an example of bad faith, right? So the question is 
what technology enables me to live an authentic digital life, right? And I think the clue is 
probably in the title. 


Dave Millard: The answer for me is hypertext lets me live an authentic digital life. Hypertext 
is a mechanism that enables us to engage with our information, build the structures and rules, 
and think the way we want to think. It's an empowering technology. Ted Nelson, nearly 50 
years ago now, said you can and must understand computers now, right? And that's because 
he was very clear that this would define the way we communicated and the way that we 
thought, right? So taking control of it and taking ownership of it was really was really of the 
utmost importance. So how does hypertext enable us to do that? Well, it is to do with linking, 
but it's what you do with those links. It's the fact that you can then use those links to construct 
other. A structures to engage in the way that you want to engage. Now SharePoint gives me 
hierarchies, which is fine, but there are other things I might want to do. And Hypertextual 
technologies enable us to engage with information in the way that we want. Now, I think this 
comes down to a principle of kind of non-regular structure. Effectively, it enables us to be 
inventive with the structures that we that we use. So that takes me to my second French 


philosopher. 
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Dave Millard: So Derrida is associated with Post-structuralism. So Structuralism is this idea 
that we understand the world through kind of grand structures. Grand sort of meanings. And 
the Post-structural movement doesn't abandon structure completely. It just says there are 
alternative structures. There are different ways of looking at the world, different structures 
you can engage. And Derrida said that if we if we enable this, if we say you can take a 
position and think about the world in these different ways, it allows you to engage in what he 
calls you Libra or free play, right? So I think essentially what we're saying about hypertext is 
it's a it's a set of technologies that enable us to use non-regular structure to to build meaning 
in our informational lives and engage in those and engage in free play in our digital 
information systems. That's what hypertext is about. And when Mark and I looked at the kind 
of literature and we saw these traditions of hypertext, we essentially see this philosophy 
applied to different areas of life. So to knowledge, to learning, to software, to interactions, 


creativity. 


Dave Millard: Sociality and play essentially. And it manifests in different ways. And I think 
what we have in, in the sort of extended reality space is a new space in which we can take 
that philosophy and that idea and try to apply it. So some people have called this hypertext as 
method the idea that hypertext is a way of thinking about the world that you can apply in 
different spaces. Now I've probably left Mark about one minute to say a little bit on Run the 
exciting part. I have run the exciting part. Shall I just I just want to so I'm not going to go 
through these areas in detail. I don't have time, but if you're interested, we can we can point 
you at where the paper is. But essentially there's a long history of systems that engage with 
knowledge and learning systems which look at hypertext as a sort of software infrastructure. 
You're talking about interaction, so different types of interactions. And of course in the more 


sort of social and creative spaces, how hypertext manifests in those, those kinds of ways. 


Dave Millard: I'm very struck that we actually see what sometimes be called feral hypertext, 
hypertext in the wild. If we had this explosion in the Web two era, but we see a similar 
explosion today. So we see people talking about human in the loop for AI. We see a new 
generation of personal knowledge, management, tools and systems. We see AR and VR and 
people exploring what it means in those kinds of contexts. This kind of new canvas that 
Frodo was talking about. So we clearly are at a point where there's an opportunity to take this 
hypertextual thinking and apply it in a new space. At the same time, there are things that have 
been left behind. So hypertext is not just about links and link following and those kinds of 


structures, but more ambiguous structures like spatial hypertext, spatial, hypermedia. There's 
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this whole idea about how it interacts with artificial intelligence or adaptive hypermedia, how 
it manifests in hypertext literature and games. And this idea of, you know, hypertext doesn't 
kill people, but people kill people. We have problems in our hypertextual spaces. So we have 
social problems. We have problems around kind of bad actors and how we defend our 
technologies and defend ourselves in those spaces. And those are things that we have yet to 
apply in these these kind of new areas. So in conclusion to this idea about what is hypertext, I 
would say that in our in our kind of survey, we really see that hypertext is a mindset rich with 
virality, by which I mean we're not the only ones talking about this, but there are people out 
there in the world actively pursuing this and you see it manifesting in many different domains 
in many different spaces. So hypertext is very much alive, very much still has things to say, 
still has lessons for us all to learn. And I think the question for this community is we can see 
some of these connections that others can't. So how do we reach across the divide, help 
people make those connections and kind of learn from our from our shared history? Thank 


you very much. That's probably more than five minutes. Apologies. 


Frode Hegland: Any questions? 


Leon Van Kammen: My question is, do you think that hypertext is a hypermedia or a form of 
hypermedia? In other words, is hypermedia in hypertext and is hypertext and hypermedia? I 
hope that makes sense, but I think so. I won't repeat that. And this will also be later in my 
presentation. A sort of question I have is that it's a bit confusing. If you look at this 
hypermedia which was added later, whether yeah, it's a bit confusing what came first and is 


this separation even a good separation? Yeah. 


Dave Millard: So I guess the quick and trivial answer is that generally I think people in that 
space don't don't make a distinction at all and they use the two words completely 
interchangeably. So when I'm talking about hypertext, I am talking about hypermedia. The 
very, very early systems were purely textual. But that's probably a limitation of the 
technology of the day. But yes, I think, you know, quite clearly, it doesn't have to be textual. 
It's about information presented in whatever form. I think you sort of suggested reducing it 
down to sort of symbolic logics, I think. And it's and it's just, you know, information that is 
presented in a form that's suitable for for people. So perhaps perhaps that's the way to think 
about it. It's kind of information for people, you know, goes back to Doug's ideas of 


augmenting augmenting human intellect. 


Brandel Zachernuk: Really exciting to to use this as a as a frame of reference. My question is, 
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um, obviously, you know, just, just presenting these, these lenses as a descriptor in sort of 
mechanism to characterize the disparate uses of, of hypertext is an important kind of 
foundational work. One of the questions I have, the question I have is about, um. About the 
utility of editors and authoring as applied to these these lenses. One of the challenges that sort 
of besets word processing is that when a word processor fails to clarify and specialize its 
function for anything beyond the idea that it's just words, you know, you just need a word 
editor for words is that you can't have further opinions about what those words are for. And 
and so you can't refine the sort of the function of it when you take this lens and see people 
sort of making use of the sort of nominal hypertext authoring environments that exist today, 
do you see frictions and fluidity that apply when people are sort of sort of using and 
alternately misusing the tools in terms of what the what the specific authoring environment 
believes is its primary function? Like, is that like as a next step, like when when applying this 
lens, like does it help tell you what should be included and excluded in a given authoring 


environment? 


Dave Millard: So I'm guessing actually you have something to say about this because you've 
been involved in sort of hypertext authoring and maintenance for for a long time. I So I'll 
hand over to you in a second. I think that there's. I think what it tells you is that, first of all, 
that there are different domains and different different ways of thinking about it and different 
ways in which you might want to support people in different spaces, but also that there are 
probably common challenges that come up and that the solutions to those challenges might 
then manifest in different ways. I know Mark has been involved in hypertext authoring 
hypertext, looking after large scale hypertext for a while, so I'll you have something to say 


about those spaces. 


Mark Anderson: Um, I think as, as we said, we've taken the view that although we, although 
we've sort of given various aspects of hypertext there really to as much as anything to help 
those with, without familiarity of the field see what it is, I don't think, I think it's difficult for 
the, the tools themselves to have too much opinion. I mean, I very much come to see 
hypertext just as a way of thought, and I think it's probably a non-normative human 
behaviour. A lot of people actually generally crave more rigidity and more structure and 
hypertext is innately more freewheeling. But it's not a it's definitely not a binary point, but I 
think most people probably are less comfortable with hypertext than those who do it a lot. 
Um, the you raise a very good point. And one of the things when I'm sort of helping people 
with hypertext tools is, is explain to them that in a sense they need they need to be sort of 


present in the authoring process. So it's not a matter of the tool telling you what to do. 
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Generally, we're educated to use especially software tools in the manner that the manual 
prescribes, which is unhelpful for this sort of exploratory and very interconnected work. And 
it's problematic too. And you can sort of see we're going through a new loop of it with AI at 
the moment where you have a magic eight ball that tells you what the supposed answer is 
when really what you were trying to do is ask a nuanced question and you wanted to know a 


range of the things that were there so that you could then better employ your knowledge. 


Mark Anderson: Um, so I don't think the rules are set here. I think you raising very good 
points. And what that really speaks to is that those building tools want to leave space for this 
sort of exploration. There's absolutely nothing wrong with the thing that just writes that short 
list or does your shopping list or something. But that's a rather different sort of word editing, 
text editing tool. So it's definitely another case that in terms of editing text and things, it's not 
a one size fits all. And I'd also just throw in a link to something that when I think about where 
spatial hypertext has been and where it might go, I very much think of tools like Ulis Space. I 
think if I get the name right, that's very much where I think the extra environment of AR and 
VR can help is basically extended thinking space and freed from the linearity of the paper 


page. I'm not sure if that's answered the question. 


Dave Millard: So just something to add to that. So one of the one of the useful things about 
associating hypertext in sport is sense with, say, existentialism is it lets you think about some 
of the criticisms that were leveled against existentialism and they're therefore relevant 
criticisms for for us. So one of those criticisms is this this sort of it's a very frightening idea, 
the idea that actually there isn't a predefined way of working, that you, in the case of the 
philosophy through life and that it's something you have to build yourself and it's daunting 
and difficult for people. So I think some of the. 


Dave Millard: So I think one of the challenges we have is, is dealing with that and how we 
we help people cope with that freedom and cope with that, that that liberty that the 
technology offers them. And some people do find that off putting and find it difficult. 


Fabien Benetou: Wonder about my biases. VR, EX, when we switch from hypertext and 
hypermedia, I was thinking when we have a virtual world and directing that virtual world, we 
can transclude another virtual world in it. So I have like a small version of it. So we it does 
apply directly today to going and reaching through virtual worlds one to another, translating 


different 3D environment content and even people in it. And I just never thought about 
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Transclusion in in exile before. But that still makes sense. Like with the latest, let's. 


Dave Millard: My current research area is, is is mixed mixed reality systems. Actually, it's 
something that I started with Dave back in the Equator Project back in the early 2000. And 
when we first came to mixed reality systems, we were hypertext people. We had a history of 
doing hypertext work. So it's that old adage, you know, to someone with a hammer, 
everything looks like a nail. So to us, people said, oh, know, mixed reality. And we said, Well, 
that's just physical hypertext, right? Literally, it's information overlaid not over textual 
documents or video or images, but just overlaid over physical environments or virtual 
environments or vice versa. So yeah, so I think it's it's a way of thinking you can apply it. It's 
not the only way of thinking about those environments, but it, it gives you some advantages. 
And, and one of the things that's done for us is it's highlighted some of the limitations and 
deficiencies of current SR systems because they don't take those relationships that far. And 
what you're saying is an example of how you would push it in the worlds within worlds, 


layers upon layers. 


Frode Hegland: One of the things that I thought was great about your paper that I really want 
to highlight for everybody is that hypertext as a tool for thoughts and hypertext, as a tool for 
presentation. It's not the same thing, which is why it's easy to make this amazing thing while 
you're thinking. And then when someone looks at it, it's a complete mess. It's the whole point. 


So that was a huge liberating thing. So thank you very much, guys. Thank you. 
Transcript: Nayana Prakash 


Hi everyone! Hopefully you can all hear me online and in person. So completely different 
presentation and it will be not that long because I don't actually have a lot of answers. I just 
have a lot of questions. So my name is Nyana. I'm a PhD student at the Oxford Internet 
Institute, which is a department at the University of Oxford that focuses on social science of 
the internet. But my PhD is about digital storytelling, and this presentation is about 
something I've been thinking about a lot, which is not text, but aurality and the internet. So 
what exactly is orality? Well, it's just speaking, really. But specifically what I'm thinking 
about in relation to my PhD is my PhD is about digital storytelling in India and oral 
storytelling. I mean, I'm going through this very quickly, but also oral storytelling is a key 
tradition in a lot of Indian cultures and across Indian languages. Um, it's, you know, a method 


for preserving history, for recording the experiences and stories of various communities. And 
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it's very audience specific. So the idea of in-person, digital or in-person storytelling as 
performance is that the story is told again and again, orally but not written down, such that 
through repetition, audiences and individual listeners kind of understand what the story is 
about. And it's also very reliant on the space and the environment that it's in and the 


individual audiences. 


Nayana Prakash: You know, the story doesn't stay the same from telling to telling. It changes 
a lot. And more recently and you know, yeah, as I said, it's impermanent. It's changeable. It's 
not locked down. The whole idea is that it is meant to be in flux. And something I've been 
thinking about is the, you know, orality and the internet. Okay. Well, I have been thinking 
about it, but it's not appearing on screen. Yes. Okay. Um, so recently in India, in the last few 
years, there's been more of a push to archive and to digitize aspects of oral storytelling. So 
this is an example of a, um, a map where they've collected histories and stories relating to 
partition from 1947, which is when India and Pakistan were divided into separate countries. 
This is a really traumatic moment in kind of South Asian history. And there's been this 
realization in the last decade or so that a lot of the people who were alive to witness and to be 
part of partition may not be with us for much longer. So there's been a kind of idea to collect 
these kind of oral stories. And, I mean, this is just one one really good example of ways in 
which that has been trying to happen. And obviously in countries like India, where literacy 


levels, perhaps written into literacy levels aren't as high as they are in, for example, the UK. 


Nayana Prakash: And there's the idea is to collect stories in a way that is authentic to the 
people telling them and makes people able to share their experiences in a variety of forms. I 
think in non archival use, we've also seen a growth in things like like WhatsApp voice note 
function, which is actually incredibly popular in India because it allows users to bypass the 
necessity of literacy and also the necessity of speaking in English. You know, it's possible to 
use 1f you're bilingual or multilingual, you can use the voice note function to speak and to 
resist the, you know, boxing in of language and of speech actually altogether, because 
obviously, voice notes capture more than just voice. They capture all kinds of sounds. And 
WhatsApp in general has a really widespread usage in India. And the voice note function is 
definitely a part of that. Um, okay. So broadly, I mean, this is a really big whistle stop tour 
through that. But essentially what I'm thinking about is how orality and orality in the internet 
reconfigures intimacy and space and closeness. So right now I'm presenting to everyone and 


I'm sort of saying things depending on who's here in the room. 


Nayana Prakash: I hope for the people online that kind of came through a little bit. So 
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essentially I was saying, how does morality reconfigure space, intimacy and closeness? 


Nayana Prakash: And I was wondering also to what extent is orality the future in any way of 
communication on the internet? I've noticed in kind of looking into the way orality is used as 
an archival tool, or to think about the ways in which orality is a kind of state of being and a 
state of communication that we've perhaps left behind, certainly in many parts of the West. 
You know, Walter J. Ong talks about this a lot in his work about orality and writing as 
technology. Is that true? Orality is quite rare. You know, the idea of true orality being more 
than quote unquote, I think I think it's textual taxidermy where you just bring things to life 
that are written down, you know, true orality is thinking without writing for all of us in the 
room. When we communicate, we communicate thinking about words. So how does orality 
change? You know, for a truly oral culture? So is orality linked with a sense of past or does it 
have some kind of future, you know, necessity in communication as well? I'm really 
interested in specifically these moments of orality and sound that go beyond language and 
beyond a specific language, and allow the kind of resistance of a uniformity in 
communication on the internet. But I'm really interested to hear anyone's thoughts. Questions. 
I know it's quite a it's been a bit of a jumpy presentation, but yeah, I think that's I think that's 
it for me. Thank you. 


Dave Millard: Dave Millard again. Um, so I was just thinking about sort of oral storytelling 
and one of the, the interesting things for me about oral storytelling is this interaction word 
again, is it's interactive. Yeah. And we've got very used to stories that are sort of ossified. 
Yeah. And linearized through novels and film and so on. But oral storytelling has this kind of 
interactive component. So I just wondered how important was that and whether, whether you 
feel that that manifests in, um, internet storytelling in a way in, in non non interactive 


technologies. 


Nayana Prakash: Yeah. No, I think that's a really good point. I think the, the dynamic nature 
of storytelling and that's what I was talking about with it is audience dependent. You know, 
you look at someone's face, you see how they're responding and you change what you're 
saying. You change, you know, the, you know, an audience of children versus an audience of 
adults. And there's certainly examples that I don't have time to go into now, but of stories 
being kind of flattened for a more generalized audience because of this ossification, because 
they can't be audience specific anymore. And also this, you know, a kind of storytelling that's 
meant to be experienced live is can now be experienced asynchronously. Asynchronously. 


Yeah. Um, so and that, you know, the idea of timing and delay also I think features in 
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storytelling as well. We're not seeing things. You know, we're seeing something that perhaps 
the storyteller made months or years ago in, as we all know, our emotions change a lot, even 
on a kind of daily, hourly basis. So yeah, it is a sort of ossification. But there also, as I hope I, 
you know, showed huge benefits to this idea of capturing some aspects of a reality that we 
might otherwise lose. 


Speaker2: What we're going to do. 


Frode Hegland: Dave, you move over here, please. We're going to use this as the question 


chair. So I'm really sorry, but that's the only way people can hear. So you have a question? 


Speaker2: Yeah. Move it up. That's good, because. 


Nayana Prakash: I just don't have to move around too much. It's very intimate. Yeah, I know. 


Anne Allen: So I had two thoughts. I find this really fascinating because 100 years ago in 
Europe, this happened where lots of recordings were made of folk stories and and also music, 
like folk music, but it was very much as an archive. And I've been really intrigued as how we 
could bring this forward and build on it in the future. And it relates back to a previous talk 
about hyperlinks, because you can't do hyperlinks in recordings. Yeah. And so just personally, 
I've realized I've got hundreds of WhatsApp recordings, memos, and they're all just archived 
in date order. Yeah. Whereas I want to use them for the future and but I can't hyperlink them 
all. So I don't know how to find each one. Yeah. And it's I guess there's no big question about 
that. But how can we use the oral recordings that we have in a way that means that we can 
immediately find the one that has the right keywords, if you like, rather than just they're all 
stored in date order on your phone or on your laptop. 


Speaker2: Sorry. 


Nayana Prakash: Um, I almost wish I had better answers to that because I'm not really a 
person who builds stuff. So if anyone has a great software or something that does that, please 
feel free to come forward. What I was going to say is that it's interesting what you talk about 
with archiving voice notes, because I think there are two kinds of archive that I've ended up 
talking about. One is this very deliberate, trying to, like, capture people's voices almost 
before they're gone, the deliberate archiving of history. And then there's this sort of accidental 
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archive that we build up when people send us voice notes and when we collect, you know, 
and actually, there are, I think, privacy considerations. And, you know, an interesting thing 
that someone sends you a voice note not really intending for it to be kept on your phone for 
years and years, but obviously you do have it for years and years and this sort of 
unintentional archive that we build up of other people, I think, is something that is a really 
interesting representation of daily life. Um, and I don't really have answers to how we can 
better link between it, but yeah, if anyone does, I'm so sorry. This isn't a very good response, 


but I was just thinking about this actually is a very bad response. 


Anne Allen: So we thought we need to create. 


Nayana Prakash: Yeah. So if anyone but if anyone immediately is like, obviously you should 


have said this, now is a great time to say that thing. 


Vint Cerf: So it's Vint Cerf. Yeah. Let me just start by reminding everyone that many people 
think that the invention of language is one of the most powerful things distinguishing humans 
from other mammals in the world. And subsequent to that, of course, writing your mention of 
orality immediately made me think of something different from what you presented, and that 
had to do with dialogue and interaction and the question of the role of dialogue in the future 
of text. What we're seeing right now is a dramatic improvement in the ability of computers to 
carry on dialogue in a way that's semantically useful. And so I'm wondering whether you 
thought a little bit about how we interact with content, which could be other than text, I 
mean, images and everything else with dialogue, and does that show up in the pantheon of 


orality or you're primarily focused on the storytelling side? 


Nayana Prakash: No, I'm interested in orality in a wider sense, and that's a really good point. 
And I think what I mentioned earlier about the asynchronicity is an interesting aspect of 
internet orality. Someone sends you something, maybe there's a break and then you can 
respond to it in a way that's quite different to a lot of human face to face interaction, where 
you have to respond right away. You know, you can't just respond to it when it's convenient 
for you. And I think this relationship with time really changes the, you know, nature of 
orality. Like, you're not like what I talked about with primary orality, where you've almost 
written a response in your head before you get back to someone when they, you know, with 
the internet. And I think that in person we are much more forced to make spontaneous right 
now what I'm doing, like spontaneous answers or responses to questions. Um, so I think that 


is a really yeah. I'd love to hear more about any of your thoughts about dialogue, actually, 
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because I think that is something that's often missing, I think, from thinking about orality in 
the internet, because it is so often not a dialogue. It's a one way stream of orality as I. Dave 


was saying. 


Vint Cerf: Of course, the primary point of the internet was to allow for the interaction to 
happen. We wanted things to interact with each other across this network. Let me just add one 
other thing it has to do with the hypertext presentation that came up earlier. The term itself is 
self-limiting in a way, and what we want to do, I think, is to extend this to hypertext for many 
values of x. So I wonder what orality, imagery, databases, videos, all these forms of 
information could have associations and connections and links. And so an awful lot of the 
hyper X world is how are our expressions connected to each other. And the idea of being able 
to use an oral interface to explore those connections is very exciting, because now I can have 
a dialogue with a system that knows about all those different media, and can allow me to 


explore what the connections are between them. 


Nayana Prakash: Yeah, that's such a great point, because I think what I'm saying is we're not 
trying to replace other forms of communication with orality. It's trying to build a fuller way to 
communicate with people. That allows people for whom morality is the primary way of 
thinking and communicating, for that to be their method of communication on the internet. 
You know, it's not a thing that we're going to see. I'm not saying we're going to see social 
media replaced by oral kinds of form of communication, but to allow. Oh, as wide and as 
broad as you know, a sense of communication and dialogue is possible in all the different 
languages and non-linguistic forms of communication that people across different cultures 
have, which I think is something that we aren't very good at encouraging even in, so, you 
know, face to face interaction. But it would be great if the internet could do that. And I think 
we we have some of those capacities and capabilities already. There is a question online, but 


if you want to move. 


Frode Hegland: There is a question and we'll absolutely take it. Even though it's Brandel 
again, it's never a bad thing. I'm just going to stop sharing because that's a bit useless right 
now. We have to go back on sharing. Yeah, I think. 


Brandel Zachernuk: One of the features of orality, it strikes me, is that as well as being 
primarily oral and audio, it is also a system where the recapitulation of the message, the 
reproduction of what what the message is understood to be happens at every time it is retold 
in the book. In the book Too Much to Know, they talk about the fact that early literacy needed 
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to be transcribed every few decades or so. But that's necessary. But that's still a period at 
which the message has the opportunity to be rechallenged and and reconsidered in terms of 
whether or how it should change. And various religious practices have put down edicts to 
make sure that it remains durable. I'm curious about what an orality that is supported by such 
a bafflingly enduring substrate as digital media turns out to be, because orality, as we've 
understood it in the past, has been necessarily transient, whereas messages that are spoken 


now are very enduring. What are the differences that you see from that? 


Nayana Prakash: Yeah, I think that's a great point. And kind of hopefully what I was talking 
about with this tension. So just the question was about how orality has historically meant that 
a message is repeated again and again, but it can change time to time, or a story can change 
time to time. And that's allowed. And that's part of the formula, so to speak. But now, you 
know, the digitization of orality means that things are captured. I think even the word capture 
is quite interesting in that sense. It can both mean to, you know, to get something before it's 
gone. But also capture suggests a kind of pinning down and a domestication of something 
that's meant to be maybe living. I think maybe, as I said before, I should be. I don't think this 
is going to be a primary or only method of listening to people or hearing people. You know, 
we can hear people in lots of different ways, and this is one of them. It hopefully doesn't 
mean that all, you know, oral storytelling becomes digital. I don't think it will do. I think just 
as all storytelling is not, you know, print or oral storytelling is not digital. You know, all 
orality is not going to be digital. So I don't think it necessarily impacts the practice. But it 
does, I think, create a wider question of what is like, are things permanent that perhaps were 
meant to be impermanent? Um, and certainly in religious capacities. So the Vedas, which are 
like the Indian, the Hindu historic texts have always primarily been passed down orally. It's 
only quite recently they've been written down. And the belief has always been that in 
communicating them orally, that's where they get their strength and and that in chanting 
them, that's where the religious meaning of the text is known. Um, yes. Sorry. So I'm going to 
finish there. But yeah, it is a it is a tension, I think, and I'm very happy to talk about it some 
other time. Thanks. 


Transcript: Vint Cerf 


Vint Cerf: That's right. I guess I need to sit as close to the microphone as possible, so I will 
do that. This is going to be purely kaleidoscopic, because I want to deliver a bunch of little 


ideas to you that are not necessarily fully connected. And I'm hoping that even after I exit, 
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you'll consider exercising your imagination kaleidoscopic. 


Vint Cerf: So first of all, just a comment about the notion of future of text. I immediately 
thought about what's the pasta of texts? And that must be alphabet soup. So one bad pun out 
of the way. Um, another thing we talked earlier. I mentioned hyper x for various values of x. I 
want you to think what is hyper sound? What is hyper imagery? Does it mean anything? 
Should we want it to mean something and think a little bit about the way in which hyper 
content is created? Sometimes it's created by an author who wants to use the hyper capability 
to steer you, to offer to you associations of things. But on the other side of that, the consumer 
may want to have tools that allow exploration of associations that were not intended by the 
writer, but which might be afforded to you by the capacity of the system to find associations 
for you, help you, help you discover them. So those are two different aspects of the creation 
and use of hyper X. One of the things that I referred to earlier about Frodo's attempt to get 
tools into the hands of people, and the resistance that he encounters. This is for text creation 
for the text use, for example, the visual meta component of PDF, which is helps us use the 


text better than we might otherwise. 


Vint Cerf: The tools that that he's developed offer a reader and visual meta. Are all require 
skills to use effectively. And if you watch Frode doing a demonstration using. Keyboard 
stroking. It's like playing a musical instrument. And in a sense we should be concerned about 
that in the following sense. To use these tools effectively, you have to learn how to use them. 
And it may be just as difficult and compelling as playing a musical instrument. You use these 
things, you know, 10,000 hours worth of use have given you skill that others might have 
difficulty achieving. You don't want tools that are so hard to use that people will be put off to 
use them, even though when you can see them being used, they're enormously compelling 
and powerful, just like, you know, Pablo Casals playing a cello. So that's not a criticism. It's 
an observation that makes me wonder, how do we make it easier for people to use the tools 
that that would benefit them? Um, one thing that immediately occurred to me about the 
generative AI comments and storytelling is that the speed with which the generative Al 
systems can produce text or imagery, or perhaps in the future sound and the like might allow 
you to tell a story in real time with the help of the generative AI. Which is kind of interesting. 


I mean, illustrations could pop up and so on. 


Vint Cerf: We already know people are composing stories using the generative. I just might 
not be quite in real time, but you can easily imagine that happening. The other observation 


about computer based tools that we've been talking about for all these years, what our 
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primary focus has been on organizing, assisting and communicating thought. That's really 
what these tools have been for. An interesting observation about text in particular. Writing in 
particular, is that when you write something down, it allows you to go back to see what you 
wrote and to make corrections and changes. And so there is a time component which I find 
interesting about the act of writing. It's just like computer chip sets. If you write, if you 
design a computer circuit that has no memory in it, you push stuff in at the top, something 
happens to it, and something pops out at the bottom end of story. When you have memory in 
a computer based system, suddenly an enormously larger amount of functionality is available 
to you. The same argument could be made for the the substance of writing. The consequences 
of writing. Last point I think has to do with the chat GPT stuff, which of course has taken the 
world by storm. And there is in fact, it's a new tool. We could argue that it's a quirky tool, and 
it does things that we don't intend necessarily or don't necessarily want, but there is a 


discipline that's starting to emerge called prompt engineering. 


Vint Cerf: And if you've used chat bots, you know that that you give it a prompt and there is a 
window. This is a technical term window in the chat GPT or chat bots. The window is how 
much prompt material can it handle? The more it can handle, the more interesting the results 
are potentially going to be. So prompt engineering is a new tool, like a pen or a pencil. It's a 
way of generating text with the help of of the chat bots. So. That I think I'm going to call that 
conceptual calligraphy, because that's kind of what's going on. So I'll stop there. And I'm 
looking forward to learning what you discussed later on today. I apologize for not being able 
to stay, but I have to summarize two days worth of Royal Society discussions on science in 
the interests of justice. And I can tell you that the non-trivial task and I've missed the 
morning, which means that I'll have to make things up to summarize that part. So let me 
thank Frodo for his persistence in organizing these things and his determination to make 
these concepts real and useful, and to, of course, provide endless entertainment for all of us. 
So thank you, Frode. 


Frode Hegland: Thank you. 


Vint Cerf: Thank you very much. 


Frode Hegland: Thank you. Have another minute. 


Vint Cerf: Do you want to sit here? No, no, no. 
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Frode Hegland: This is good. This is good. Somebody take a picture. All right. Press the 
button. No, no, Keith. Well, right. Okay. So here this is really I think today is very, very 
special. And it's got a lot to do with the last two presentations. And you kind of chewed out of 
that in a way. And that is so yesterday some of us we were having a barbecue. If you weren't 
invited. Sorry. It's because I didn't know you were here. Right. But we'll have more 
barbecues. And I was outside with Leon and I burnt my hair. I burnt a thin hair on fire. Hair 


on fire. 
Vint Cerf: Well, we knew that about you all along. 


Frode Hegland: But there is a point to this. So I burnt a thin line of hair. And for the entire 
reason, no evening. Nobody noticed. Right. No, no, no, I was brushing my teeth. I was like, 
what in the world is that? Even my wife didn't notice. And for this weekend, I've really been 
trying because I'm editing my pieces. I've been trying to look at a lot of text. I have a big, 
beautiful display. You want to help me get it? Because you say the air is almost like a display, 
so the human eye is not trained to stare. Like I'm looking at you now. That is abnormal. 
Unless there's a really good reason we don't do that. Our eyes dart around a lot. Right. So the 
fact that no one noticed this line of change made me really realize that. So we're talking about 
audio and different forms when you're talking about all these different aspects. I'm so grateful 
for that because finally, in history, we have an opportunity to actually build them together. 
Right. So your kaleidoscope is wonderful, and I hope it can have that as the theme of today to 
try to have more kaleidoscopic thinking. So thank you very, very much. 


Vint Cerf: You're very welcome. 


Transcript: Mintter 


Gabo Beaumont: Thank you very much. Everyone too and Frode, to have us here. I'm Gabo 
Palacio. We are from Minter and we are going to present our what we have been working on 
for the last almost four years. Um, so basically, as I discovered hypertext four years ago, and 
I was very struck with the whole concept, and I was very surprised that it's not a more 
common concept in the, in human society as it is. It's it is our medium for thinking and 
communication. And that's big and that's huge. And we are fascinated with with it. Um, and 


we also discovered that the web as a hypertext system has very deep technical limitations, 


oo). 


um, making it open for publishing. So you set up your web server, you publish, that's 
decentralized, that's um, um, that's permissionless and that's great. The problem is that for 
collaboration, you actually need to it's not decentralized. We actually need to go to all the 
central platforms, all these central databases. And and that's because of the technical 
limitations of the web as a hypertext. And the problem with all these central platforms is that 
it's like central planning. They are not efficient enough to handle the complexity of our digital 
societies. They are not. So many of the problems we are, we are getting is because of because 
of that. Central planning doesn't deal with Vale good enough with. Um, the dynamic, our 
dynamics and complexity of our societies. So what we are working is in an open hypermedia 
system for the web, and with the vision that intellectual communities should be the one 
running the internet and not all the central platforms. Um, by syncing computers. Um, so we 
have. Bill. The. 


Gabo Beaumont: So we have built this, um, a, a system hypermedia by Minter. Um, which 
involves an application that you download in your, in your, in your device. So. So it's a it's an 


application that is local first. So no servers. It runs offline. Um. 


Fabien Benetou: Like. This is the transcript. Oh, here. I'll move it on. To move it away. 


Gabo Beaumont: Nope I can't. Yeah, I do here. 


Fabien Benetou: Yeah. 


Gabo Beaumont: Okay. So I'm going to I'm going to demo. We are going to demo me how it 
works. So we have the, the local application that is in the, in the computer. And also you can, 
you can run a mentor site which is going to be on our domain. So here we have hypermedia 
hypermedia which is a domain. Or we have oops gabberts which is another domain. Um, also 
we have uh oops here here we have h h g dot link which is author's domain. So basically the 
minter we have. So we are, we are in a in the in the application. Now I have different groups. 
So for example I have group friends which is Gabor. This is my personal website. So I can 
invite different members. Everything is everything is cryptographically signed. So basically 
adding someone to a group means that you are adding his public key to the to the to the to the 
site. And and he's going he's going to be able everything that is published is going to be able 
is going to be part of the of the of the collection of content of this, of this group. So for 


example. 
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Gabo Beaumont: So you can create. So it's a hierarchical. It's a hierarchical. Um. Um editor 
multimedia where you can add different, you can add different type of media, images, audio, 
video, you can add files, PDFs, um, and whatever you have, you have contacts, which is 
direct connections with peers where the, the, the content is going to be synced depending on, 
on the people that you are connected with. Um. And so here is a ratio that here is all his 
content. These are his groups. Um and so for example history of hypertext. This is a 
document I have published to my site. So if I go to my to, to to the site which is gouge's 
history of hypertext. So here is the, the, the content that I've, that I've created with my 
application. Um, everything is digitally signed. So here is, that is me. This is a public key and 
anyone can verify that this text is mine. Um, keys are digitally signed. Content is so 
important for a distributor to make in a distributed system, because then you don't need a 
central databases with login password. And this, this is this is you can manage identities only 
with digital signatures. No blockchain, no nothing, only digital signatures. Um, so in this 
case, in this in this document, there has been contributions by Gabo and by Orazio. And also 
we have reused content from another document, as we may think here. Um, the documents 
are version control. So you can you can see how the alchemists evolve. Um, it's very inspired 
on git. So it's very version control. So I can see I can, I can see how the document evolves 
with time to the next version, to the next version, to the next version. And I can see how all 
the content starts adding up. Um. And so, for example, if I go to to the document so I can I 
can go to create a new or maybe you want to do it. So basically I have my local node also 
running, and we are peer to peer connected with both the site that the group and his machine. 
So any change that I do to those documents will be propagated to all the nodes that are 
connected. So essentially everyone will know that there's a new version of that, or there's a 
new change that creates a new version that you can see not only on the site if it's published, 
but also in any peer that is connected. And because everything is immutable, everything is 
cryptographically signed. Everybody will know when it happened and who did it, because 
I'm adding my public key on that immutable data. So if I do, for example, I edit well, I'm not 
sharing right now, but I'm in that document. I'm going to do version hypertext at the end. 
Right. 


Fabien Benetou: So if I. 


Horacio Herrera: Commit this change to Gabo's friends, which is the group that I'm part of it. 


And I committed. It will be available in S when he reloads. 
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Gabo Beaumont: So the interesting thing is that we are getting we are bringing immutable 

content because here I'm in this version of the document which is a hash of the content. So 
this is completely immutable. So if I refresh I won't see his change because this is an exact 
version of the document. I'm not. It's immutable. And this is so important for having a non 
breakable links. So if someone has a link to one of these or any part of this text, that link is 
not going to break forever. Because another thing is that when you link once you link to 


something. So if I'm in, well first I'm going to go to the last version. 


Fabien Benetou: Um. 


Gabo Beaumont: I'm going to I'm going to the to the last version and I'm going to to see here 
version hypertext. That is the last content that his computer has, has published and his device 
has signed and that I can verify that this piece of text of the whole document is his and 
everything without central server. I mean, this went from this computer to this server, which I 
set up in DigitalOcean. No Mint.com or whatsoever involved here. Um. And so links. So for 
example, if I'm in a in a in another document. So for example we have a history of hypertext. 
So we are missing something about Project Xanadu and Ted Nelson. What we can do is we 
can transclude we can transclude paragraphs. So I can copy this paragraph right now, the last 
the smallest thing we can address is paragraphs. But but because we haven't done it yet, but 
the whole architecture of the system is that we can even transclude characters. But now I'm 


going to conclude this paragraph to the. So I went to edit. 


Fabien Benetou: You. We better create a new draft. 


Gabo Beaumont: Oh, yeah, because I didn't get that last one. 


Fabien Benetou: Yeah. Now you're in the last one. 


Gabo Beaumont: Okay. Now. So boom. Just with a link. I'm and I, I created a transclusion to 
the to Ted Nelson's paragraph I commit. So here I have a transmutable. So this is this 
fragment is verifiable that this fragment is from another document signed by another author. 


So now when I go to to the web, as we may think. 


Fabien Benetou: The phrase. 
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Gabo Beaumont: I go to. 


Fabien Benetou: The. 


Gabo Beaumont: Oh, sorry. This one. Yeah. So I get this paragraph, which is not mine, which 
I brought from another, from another document, which when I click, I get to this to the, to the 
other document here in this, this is authored by me, not by TED, but it should, it should say 
we should trace if it was by TED. And and the cool thing is that when you reference 
something, you start, you start storing it to in your computer. So the content that gets more 
referenced is going to be more it's going to it's going to be more permanent because my 
device gets all the references that I'm using it and I'm storing them. So this link is actually 
permanent. It's not unbreakable because I have the two sides of the link. I'm storing it on my 
computer. And, and also I have like this is a hypermedia P2P database. I have the content that 
I'm interested in. I have it local first in my computer. So it's very easy to index all the links I 
have. So it's very easy to extract the backlinks. So. So in this case I can, I can. So I'm here in 
Project Xanadu, and what I can see is that there was a another document history of hypertext 
that has cited that has referenced this. And I can travel and I can go to the, to the, to the 


reference. 


Fabien Benetou: The local machine also with the citation minute. 


Gabo Beaumont: Okay. And uh, and then of course. 


Fabien Benetou: Yes. 


Gabo Beaumont: We are so inspired with Xanadu and with Engelbart, but in this case 
Xanadu, we can also just pay Bitcoin. So what we do is we count characters of the whole of 
the whole document and we can distribute, we can do a distribution of, of micropayments to 
each of the contributors based on whatever we want characters. Maybe you want to have 20% 
of the royalty and you pay and it's instant. It's cross-border. It goes anywhere in the world you 


can pay. Someone from Bangladesh can pay someone from Argentina. And yeah. 


Horacio Herrera: This is this is also an open question too, because right now we're using 
characters. But maybe in the future we want to know what would be the best or fair way of 


distributing this kind of value to authors. If there's any other way of measuring it, it'll be it'll 
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be awesome to to talk about. Yeah. 


Fabien Benetou: And well. And yeah. That's it. Okay, who's got the first question? Well, I 
should I should think of a question. 


Keeichi Matsuda: So I think, you know, you started off talking about decentralization and the 
immutability of what you write and using that as a foundation for being able to create, you 
know, a new web, I suppose, that allows for more of this interaction, but the immutability of 
it. I'm interested how that intersects with GDPR, especially the right to be forgotten. And the 
idea that, you know, we have this enshrined in law that, you know, people have the right to be 
able to remove data from it. Is that compatible with the system? 


Gabo Beaumont: So, so the system it's so this is public. So it's for like the public discourse. 
So once you share something, if someone gets gets it and starts forever, it's going to be stored 
forever. Because again there is no kind of central servers whatsoever. So we are not in control 
of what could happen. So we cannot impose GDPR measures. But I think that that's the way 
how it works. If you publish something to the web and someone screenshots or goes to 
Internet Archive, that's how it is. It's interesting how the less the system, how it works, that 
when you reference something you are going to store it to makes that the content that is more 
referenced is going to be more permanent. And that's kind of like a memory in the system 
where if things stop being referenced, they're going to tend to, to get for, for forgotten. Um, 
so so it's like an organism when for too for like how our mind works to let's you reference 


something, it gets forgotten. And that's the only way to really. Uh, delete something on the. 


Fabien Benetou: Three quick questions. Number one, I think. Is it okay? I think it's okay. Um, 
is the client native or is it web based? Because I saw the content was web based, but I think 
the editing is not right. Um, so that's the PDF bait, let's say. Why is it not just the web? If it's 
not, you say it's a protocol. Do you have clients developed by random people like us, or is it 
just you for now? And the last question was. Uh. I mean, there was one like, do you have 
WebEx for content on it? We have WebEx for content on it. You have because discussed a bit 


about VR before. So you're open for it. 


Gabo Beaumont: So. Well it's everything is open source. So you can that was. 


Fabien Benetou: The question I actually missed. Perfect. 
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Gabo Beaumont: Thank you. So it's in it's everything is in GitHub mentor hypermedia. So 


everything is open source. 


Fabien Benetou: Uh. 


Gabo Beaumont: With the Apache license. And, um, basically we started with, uh, okay. This 
is we want to do I mean, we are in love with we fell in love with Engelbert. I want to to, um, 
to build this, this beautiful system. But then we we actually discovered, I mean, we also felt 
that the web is very good, you know, and and the web is a hypertext we have given to 
ourselves, and, and it's not going away. And I think and we believe that we need to improve 
it. So the whole idea of our, our system is to, to bring this versioning, authentication and 
micropayments to the web. So it's very so what all we are doing is, is we are trying to connect 
it very well to the web. Right. So improving the links. So the links you have the human 
readable links with our web. But then you can, you can have, you can have all the parts of the 
content where it brings the versioning and the immutability. And so it's basically very web 


based. Um, but the client right now is a desktop application. It's not a web. 


Fabien Benetou: Why? I don't understand why. 


Gabo Beaumont: Because it's a it's a peer to peer system. So you need to be thinking all the 
time. And you need to to get access to the file system. Um, and so that thinking is it's, it's that 
continuous thinking. That's one argument that that continuous thinking is you need to have 
access to the device, which from the web web browsing, you have the local store of the of the 
browser. But that's not enough for, for for for running. What the thinking we use IPFs to run 
to run the. And the second one is like because of the web limitations, the web is very limited 
to to the the browsers are very limited to our current experience, and we wanted to 
experiment with these ideas of multiple windows of visible connections. And, and there was a 
lot of a lot of limitations of the browser itself. And it kills the experience, you know, like for 
overall for the authoring of the if you want to really link everything and, and reuse and you 
know, if you really want to have that experience of authoring, uh, we need, you know, and. 


So that's why we used. 


Fabien Benetou: To expand on this. 
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Horacio Herrera: Answer. Well, the reason why we did a simplification is because there are 
two processes running at the same time. One is a local server that is connected, that is the 
IPFs node that is connecting to all the gear, that is orchestrating all that and indexing all the 
data, and also the client side, which is the one consuming that API, which is local to the client 
side. It's what is web based. But another reason why we're using the desktop application, 
because that means that the keys, your private keys for you to offer to them are local. We are 
not putting those in any local storage or anywhere that the website or our web server or our 
web page can access to. So we found that a good solution for us right now, that doesn't mean 
that maybe in the future someone will create a client that can work on a website. Or maybe 
we can create some bridge between your local node, which has access to your keys, and a 
browser that you can actually alter content in the browser. So that's a possibility that we 


haven't explored because as you may guess, it's a very complex system that to build on. 


Gabo Beaumont: So yeah, actually it's multi-device. So you have one identity, but you can 
start signing different devices. So you can start signing your content with the same identity 
and sharing the same identity from different different devices. So yeah, the signature that this 


is the device signing the content, it's key also in the system. 


Dave Millard: Yeah. So so yeah. So first of all, just I find it tremendously exciting to see this 
kind of stuff. Um, and I said there's been a kind of resurgence in people interested in 
hypertext technology and kind of revisiting some of the kind of key ideas from the 80s and 
the 90s, but with modern technology and reimagining it. And this is very much in that vein. 
So, so thank you very much. Um, so my question is actually quite boring in comparison, but 
but I'm thinking of other places where you see this reinvention. So you see for example 
personal knowledge management environment. So things like notion which are very 
centralized and they're very much pitched at teams who want to share hypertextual content 
and, and grow things. And then you have systems like obsidian, which are local first, very 
much pitched to individuals, and you're kind of offering something which sort of is, is in this 
sort of sweet spot between the two. It's both local first and it's collaborative. But where do 
you kind of see this as a, as a primary use case? So both of those systems, they have a very 
clear vision for if you are this type of person, this is the sort of hypertext system you want to 
have. What what's your vision for that type of person for for a kind of decentralized hypertext 


infrastructure. 


Gabo Beaumont: Um, so basically, um, we like the open collaboration. So collaborative 


collaboration between strangers. All these systems normally are teams that are trusted teams. 
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And as the web when it started, and I think that the idea of open collaboration or 
collaboration with the strangers that share the same, the same knowledge goals, I think that 
there it's it's where, where it's where we are missing right now from the web. Right. Um, and 
I think that it's the most powerful thing from the web so that that open collaboration where, 
where hundreds of people can, can collaborate without knowing or trusting each other. Um, 
so that's the where where we're focused. We like a lot of anger, part of digital knowledge 


communities, and we're very inspired with that. 


Frode Hegland: Brandel you are next. But just on the collaboration thing, especially the 
question of going into environments and stuff, this is what I also hope this community will do 
is to develop open ways to do that. Obviously I'm all about visual media, which I'll present 
this afternoon a little bit. Most of you know it, but we just we don't want to have an extra 
solution for mentor that is, you know, your own package. Ideally, you know, we want it to be 
able to go into the spaces. Okay. Brandel. And then in fact, Fabian, I think we have to wind it 
down a little bit. Um, yeah. Do you want to Brandel. Are you there? 


Fabien Benetou: Yeah. Okay. I also realized I. 


Speaker7: Had failed to give a brief introduction of myself, so I will take that opportunity as 
well. My name is Brandel, and I propose and represent and discuss spatial hypertext 
standards for the W3C on behalf of the browser company. So. So it's with that sort of bias in 
mind that, that I, that I ask these questions, although I'm typically representing myself and not 
not any of those big companies today. Um, so my question is similar in terms of what is the 
what is the primary, um, sort of purpose. So you mentioned collaboration, but before you also 
mentioned payment and being able to sort of attribute, uh, for the purposes of payment. Um, 
is there. So, so that could simply be the answer. But I'm curious about what. The 
consequences of successful use of this application or framework or philosophy look like en 
masse. What? How would you know when you have one? What is the difference in people's 


behavior and actions on the internet? 


Fabien Benetou: Um. 


Mark Anderson: Can you quickly just summarize that for question? So. 


Fabien Benetou: Uh, so. 
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Gabo Beaumont: The question is the question is in this system, if it's if it's successful, what's 
the what's going to be the impact? What are going to the first, what's going to be the main use 
case and what's going to be the impact of the user's behavior that's going to merely have so 
and he mentions also that that is collaboration. But collaboration is very broad, a very broad 
concept. And, and also the relationship with the idea of payments. Um, so our, our mission is 
and it's, it's amplifying intellectual communities, um, and make the experts share their, their 
knowledge again. And I think that having a strong identity with, with of bringing authorship 
with public keys, where everyone is going to be referenced, cited, making a possibility to be 
cited and, and, and, and recognizing their authorship, I think it's going to be make more 
people share what they know and be keen to to collaborate with others. I think that protecting 
that authorship, that is how academia works. But it's a very complex technical system to go 
from an idea to a publish this. Um, I think that, um, yeah. So this system with a better 
authorship, I think that will impact more people wanted to share. Um, and in, in terms of 


payments like we use bitcoin, um, we will add dollars, euros and other. 


Gabo Beaumont: But but the system like the micro payment system that allows this, we have 
only founded the way of doing this with micro payments. Um, so it's a very niche thing right 
now. So it's hard to know how it's going to evolve. But um, we would love to, to bring a 
business. So right now media either in news either in regular media or media in, in or or 
publishing on on academia, we believe that our broken all those industries are very broken. 
So academia because someone is earning a lot of money and the distribution of money is not 
very well, um, very well done and use because and news media publishing because their 
business model is broken. And in our opinion, all these media is fundamental for 
democracies. So it's it's the, the public deliberation is should be happening in those editorial 
processes. So I think that fixing those business model, enabling those editorial processes of 
review, of review of sources of, of, of validating, I think that bringing that business model to 


all those publishers would, would solve a lot of our current, uh, fragile democracy. 


Frode Hegland: It will be a second, but we have forgotten something quite important. I'm 


going to share a screen here first. And then go to Safari. 


Fabien Benetou: And again. Yes. Know what? To ask you a question. 


Frode Hegland: Oh no no no no no questions. That's too much. I was just going to say if you 
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go to the Future of Texts website and then Symposium, there is a link on here to please 
introduce yourself. Please, please do that. I should have said it in the morning, but of course I 
forgot half of what I should have said. It's just a Google doc. There's no public or private key. 
Anybody can write what they want. Oh. It's beginning. Oh. Very good. Also, if you're 
presenting, if you want to put your presentation there, it'll make it easier later for everyone, 


not just me for going through things. So that was that. Do you want to type in? 


Transcript: Leon Van Kammen 


Leon Van Kammen: All right. So thank you very much for coming and inviting me. I'm Leon 
and I am a creative technologist. And I would like to talk about the hyper mediatic 
experience. It's an experience which basically doesn't exist because I just made it up. A hyper 
mediatic is sort of my the term I sort of created, because I realized that hyper media was a bit, 
perhaps a bit old fashioned, and I didn't really get it when I was researching it. So I thought. 
Uh, realizing its potential along the way, I thought maybe let's just add some three characters 
hyper mediatic to sort of give my interpretation of of everything I found out. And perhaps it 
should be called hyper X, as in mentioned in the by Vint, I think. So it might, it might be a 
similar thing. So I was sort of going I was experimenting with trying to redefine hypermedia 
by looking through the lens of XR, like it's really easy to go, to go from text to hypertext and 
to sort of realize what we have now on the web. But it's not so easy to imagine what, you 
know, how we could reason from XR where we have a 3D space and how that sort of follows 


back into text. So yeah, this is my experiment to reason backwards. 


Leon Van Kammen: So basically from from what we have now, what we can do now, and 
also the realities which we have now because we not only have physical reality, we have a 
sort of written reality. These things can be totally different. As you know, with all history 
writing history is written by the victors usually. So anyways, I really enjoyed this actually to 
to sort of not be locked into certain historical opinions, so to speak. So one, one of the 
questions I had like which came first, text or images. And I didn't really get a good answer 
when I started to research history. It was a bit of a rabbit hole like you can it's almost the 
same and the same with like, which came first, hypertext or hypermedia? I also ask this in a 
previous presentation, and this was also a bit fuzzy for me, how this was all presented 
historically. Historically it seems a bit like hypermedia was added after hypertext. So 
anyways, for me the answer was almost like through a symbolic lens, they appear to be the 


same, these things through the lens. They also seem to appear to be the same. So hypermedia 
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would be something you would consume in XR and not necessarily through hypermedia. 


Leon Van Kammen: I'm not sure if that makes sense. But anyways, here is again, the 
traditional sort of hypermedia explanation how it's different from hypertext. To me, I found it 
a bit weird that the hypertext can link infinitely. Every hypertext document can link to 
another one, and for some reason hypermedia is explained as some kind of like one outbound 
link, but the image itself has no links inside them to link to another image. Which is a bit 
weird because if you look at a comic book, you would imagine to load one, for example, in 
XR, XR you would imagine one big sort of title of a comic book clicking on it and going to 
the next one or something. So for some reason this didn't really happen, or I haven't really 
experienced it with hypermedia in the same way, with videos, all these videos, images, audio, 
they seem to be endpoints rather than hyper connected entities. So there was a bit weird to 
me. So, um. So through the lens of reality to text synthesis. So basically going from trying to 
describe your reality in text is a sort of synthesis. And the same as I'm trying to look in XR, 
like how how can I synthesize this XR reality to some kind of text, just like HTML is also a 
sort of synthesis of some kind of experience. Um, so I came up with this. I started to play 


with this concept, which is basically a concept from synthesizers. 


Leon Van Kammen: Like this is a really like one of the first synthesizers were basically a 
carrier signal and a modulator signal, and that when added together, creates a sort of 
synthesis which creates a new sound, something which is more beautiful and and 
recognizable. And in this, in this way, we could see the Earl as a sort of textual carrier, which 
then is modulated. It carries HTML, and the hypermedia is sort of modulating the HTML. So 
you have like this almost it's almost a text synthesizer, which then finally synthesizes some, 
some text on the screen with some images and pixels and but many times I found I got this 
feeling like it's not really hyperconnected. And now, for example, the first this sort of fan out 
pattern. This is a bit like what hypertext is today. We have just a document and it's sort of a 
fans out to maybe some pictures movies, but that's it. Like you cannot go further and you're 
sort of locked into clicking another hyperlink. So you cannot really surf images, you cannot 
really surf audio. So on the right the the two ones on the right would be something more 
hyper connected. So I came sort of I started to play with this idea that maybe the real sort of 
killer app of the web of hypertext and all these things. Maybe it's just the URLs, maybe it's 
just the textual URLs which are modulating with each others. Um, and not so much locked 
into the sort of HTML paradigm. 


Leon Van Kammen: It's almost as if the HTML is sort of locking in this hyperlinks and the 
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hyper connectivity. So yeah, I started to play with this idea. Maybe it's just a text modulating 
with text where the URL is sort of the, the metro, the the transport, it teleports are texts from 
our brain to somebody else's brain. And the hypermedia could actually be anything. It could 
also be a 3D object. So if everything would link to everything. So everything is an object, and 
we are not really thinking in terms of hypertext linking to hypertext. Then I started to realize 
that TikTok and YouTube, they are great video databases, hypermedia like, but not really 
hyper mediatic. Again, this is just something I made up because I can still not click a table 
inside a video, or I cannot click on a person in a podcast. And there's like, everything is not 
like the user curated web rings of the of the web. The beginning. No, it's sort of like 
algorithm curated. So the users have not so much control of what will be the next video, 
etcetera. So I was thinking for this hyper mediatic synthesis, which I was thinking about, the 
waves that the textual public carrier seems to be the URL. And this is basically what I said a 


slide ago. Since we on short time, I'm going to speed up. How much time do we have? 


Frode Hegland: Oh yeah. Yes. You're almost done. Okay. 


Leon Van Kammen: So so in a in a more high level way I was thinking of the URL is almost 
a sort of textual time space encoding decoding because these are if you really boil it down to 
the basics, the stuff we share and the stuff the applications share with each others to sort of 
compress and decompress everything we think or model in, in 3D modelers. So last thing, 
last but not least, I want you to get totally drunk on URLs as I have became McLuhan style. 
So I was thinking maybe the URL is the message. Maybe HTML is just a hypertext media 
format. A great one, but just a text media format, similar way the web browser is just an app. 
We have seen so many apps appear, and I think maybe we should sort of like depart. A bit of 
the web browser is sort of the app now it's just an app, the same for the Http protocol. And 
yeah, I think if we just sort of try to. 


Speaker2: Um. 


Leon Van Kammen: Take a step back. Then there are possibly very nice opportunities, 
opportunities to also look at the operating system and the apps, all the apps of the operating 
system as a sort of browser. So what my last thing, what I want want to say is that the there 
are some opportunities where we can sort of sort of get the protocols, get the file formats, and 
get the views from what browsers have been doing for years and sort of pulled them into 
more into our apps, all the apps, doesn't matter which App Store and or our operating systems 
may be on Linux distros. This would be a bit easier to do, but I think. If we look at it that 
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way, there is not so much a battle between the web browser and sort of app stores apps. If we 
just think in terms of apps being browsers and operating being browsers like that, the app 
stores can just exist. We don't have to have a battle with them. They're there, but we have an 
option to just connect our apps with each other and their data using URLs. This is a small 
video of a spec I'm working on, which is basically a spatial URLs for the URLs where you 
can basically reference, just like the presentation, before you can reference a fragment, a 
piece of 3D, and sort of I think you call it transclude. And yeah, you can basically transclude 
it into another 3D document. So maybe we should have a talk actually. And this is. Yeah, 
exactly, exactly. Thanks a lot for that. And this I think this will be the end of the video allows 
us to really use this teleporting power of the URL and this also this sort of co authorship or I 


don't know how you call it. 


Speaker2: So. 


Leon Van Kammen: Yeah. Let me go. Oh, yeah. This is by by the way, my the fragment spec 
in sort of in action where the whole 3D scene has been collapsed to an XR word graph, which 
is text. So we went from, from complete 3D spatial to text and which allows associations to 
be drawn visible links that if you click on a cube, you can see the hash cube appears in the 
URL and the visible links start to appear with anything cube related. If it's a button which 
does something with a cube, or it's an image, or it's another 3D document embedded which 
has a cube, all these things can be sort of shown the relationship. So thank you very much. 


And any questions welcome. 


Speaker2: The. 


Frode Hegland: First and obvious question. And then over today what happens with the 
domain part of this. You know domains die. So how does that function then when you change 


domain or something goes down? 


Leon Van Kammen: That depends on the publisher I this this Excel fragment spec is basically 
it's protocol agnostic. So it means it can work with any protocol. If you're very serious and 
worried about this, you have to choose something like IPFs. If you're not so worried about 
this, or you want to be more EU right to be forgotten or just basically, or if you just want to 


be more lazy, you just go for Http. You upload something to on another protocol. 
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Speaker2: Can you just. 


Frode Hegland: Stop sharing screen so we can see you okay. Thank you, thank you. 


Dave Millard: Okay. So yeah was really really good. Thank you very much. Um, so this isn't 
this is sort of more of a comment really. And to answer some of the questions you raised. So I 
think we tend to look at at hypertext today through the lens of the web, because it became 
such a dominant hypertext platform. If you go Pre-web, you find loads of really interesting 
work on this space. So the place I would go is probably open hypermedia systems, not in the 
open as in open source or open as in free, but open in that what they did is they they take the 
hypertext structure and they separate it entirely from content. And that meant that the 
hypertext systems of the 90s. So Dave's over there. Microcosm, for example, allowed you to 
take arbitrary content like a video and apply hypermedia on top of it. So microcosm that was 
91, was it 90. So that would allow you to play a video, follow a link in the video to an audio 
file or link in the audio file to a word file. Et cetera. Et cetera. Images. It was all completely 
hyperlinked, but you can still run it and you can still run it. So so really have a look at open 


hyperlink. They're really interesting. 


Speaker2: Open hyperlink. 


Dave Millard: Open hypermedia systems. 


Leon Van Kammen: Open hypermedia. 


Speaker2: But it gets. 


Dave Millard: Better. Right. Because one of the things you do if you take the hypermedia 
structure away is you can take the same content, same content, and you can take an 
alternative hypermedia structure and put it on top right. So that leads you to an adaptive 
hypermedia system, where different people can have different combinations of hypertext 


structure and get personalized hypertext views completely independently of the content. 


Speaker2: Right. Interesting. 


411 


Dave Millard: And then it gets better again. Right. Because you can say, well, link structures 
is one type of content, one type of structure. But there are alternative structures like spatial 
hypertext structure, taxonomic hypertext structure, those kinds of things. So you can have 
systems which manage that and overlay that over the over the same time or do that 
simultaneously. And there was a whole area of work called component based open 
hypermedia systems, where the components were were essentially a structure server, a link 
server, and those kind of and you dynamically combine them. And then this all accumulated, 
I think, in a movement called structural computing, which was to say, this is so fundamental, 
we should push it down into the operating system layer so that these are hypermedia services 
that are operated given to us by the operating system. So this is all work. That was kind of, I 
guess from 1990 through to about 2004 or 5. And around that time, what happened is that the 
web was just crushing this work. Right? So, so much emphasis on the web. Mid 2000, you 
get the web two revolution and suddenly some of the benefits of those kinds of systems you 
now start to see on the web. And I think that's basically the final death knell. Right. And that's 
what so when I say you see some of these ideas resurfacing, that's what I mean. It feels like 
the the web tsunami has passed and we're at the other side and able to think about these 
things again. And I'm sure Dave's got some to add as well. But just a final thought. So so 
when I look at XR environments, I look at the perfect open hypermedia environment. Again, 
I don't think of hypertext embedded in, in the in the 3D objects. I think of it as, as, as 
connections and associations, which are held separately to content and dynamically combined 
at runtime to create me a personalized, interlinked experience in the XR space. And that's 
kind of like a modern manifestation of it. That's what I'd love to see that. Right. Sorry. That 


was a lot of stuff. 


Speaker2: No, that's good stuff. Yeah. Brilliant. 


Dave Du Roure: Very briefly. That was a fantastic record. Dave Drew, University of Oxford. 
Um, that was a brilliant summary. So that period. Absolutely. We were working on open 
hypermedia. Um, I'm going to tell you the question you should ask Dave. So, so we it's 
exactly what you've been describing. It was great having the link structure separate because 
you could. Different structures, the same content, or you can bring along new contents and it 
would be linked to automotive. Amazing was working in music. I still work in music. I want 
to talk about that at some point, but that works really, really well. Where we started trying to 
tell the web community how to do this, and we wrote papers about the first web conference 
as a paper about this, but not in the proceedings, because we weren't telling people wanted to 


hear about the web, not about another system. So there's a there's a period where the open 
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hypermedia sort of discourse got somehow suppressed by the web. What's really exciting this 
morning is we've seen two presentations that absolutely have gone back to that world. So 
here's a question you could ask me and Dave. Why didn't they catch on? If it's better, why 
didn't it catch on and see what Dave has to say? But my my easy answer to that, I think, is it 
required that much more engineering. As soon as you store the link data structures separate to 
the documents, you needed some more sophistication in the systems. And there was even 
though it gave you link integrity, you had to do more work to get there. And the web came 
along and we looked at Tim's first web browser demo and we said, that's too simple, that'll 
never catch on. And the rest is history. So it's simplicity of the web and the fact that people 
didn't expect link integrity. But now what you just demonstrated this morning is with current 


technologies made with that engineering is absolutely accessible now. So back to Dave. 


Dave Millard: So so well that's exactly my answer as well. So so I was going to say that what 
people I think I think looking back on it, what people didn't realize was it's that killer app 
thing. Right. It's like so the the web did something. It did lots of things terribly right. But it 
did one thing brilliantly, which was scale. Right. And none of these systems scaled at all. 
Well, right. So the web came along and it said, actually, if you throw out all of this stuff, if 
you if you throw out a disconnected structure, if you throw out version control, if you throw 
out kind of link integrity or you get rid of all of that stuff, you can scale. And we all went, 
well, that's terrible because you've just thrown out all the great stuff, right? And the web said, 
but we don't care because now we have 100 million users, right. And I think that's the key 
thing. And and actually, your system is the first time I've seen a genuine alternative, scalable 
infrastructure to the web that has the potential to deliver all of the things that were available 


in the more centralized desktop systems of the 90s. Right. 


Speaker2: Right. 


Dave Millard: So, but my my concern, and I suppose part of my question and part of my 
concern. One of the things behind my question was, was how do you deal with the 
commercial and social reality that we now have the web and the walled gardens and the apps 
and all that kind of stuff. And one of the interesting things about the XR space, I think Frode 
made this point as well, is it kind of gives you this fresh canvas where we don't have all of 
these kind of embedded interests anymore. And you could say, actually, there's the space to to 
do this in a different way. So I think the kind of combination of these and in that space that's 
that's has really exciting potential. But yeah, fundamentally it's taken, you know, 30 years to 
see it. Right. And there were attempts I'm thinking like the distributed link service and 
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playing around with proxies. And there was the the dubious work around X link and all that 
kind of stuff. Right. But it just kind of said overengineering it didn't it didn't it didn't have the, 
the simplicity and the the understandability of the web. That's my how. 


Speaker2: Interesting. 


Frode Hegland: I have comments and questions too, just so you know. So. When just 
beginning on the whole thing with the links. Doug Engelbart system, as you know, was very 
constrained, very organized, very hierarchical. And then we started discussing the idea of 
using the links to put commands into. So we did things like the view spec would be in the 
URL. So to see this not extension but this vastness of using the link. I mean, when you have a 
node with a graph with connections and nodes, you could always reverse it to what's the 
connection and what's the node. Right? I think you are really putting something very 
interesting on that. And I also. Just want to highlight the super importance of a addressability 
and b things, being able to say what they are. So the thing that I've been working on that you 
know, super well, visual meta, it just says this is a thing and this is what it is. That's all it is. 
And that allows any system to come in and use it. Right. And then the other thing when you 
were showing the the collapsing of the addressability. So I've stopped the clock. Kevin has 
left the room now he's going to be off to lunch. But think if all of us can think a little bit more 
on this notion now before lunch, because this is an opportunity to bring voice back. This is an 
opportunity to to really, like Dave just said, use the XR as an excuse to rethink things. I really 


hope someone else has some questions or comments. 


Eric Vicenti: Well, this is this is really tied up. With what? This last sentence that you just 
said is really tied up with what Vint said before, that when they are creating the internet, there 
are no commercial incentive. So they had a pretty easy to do what they did. And right now, as 
I'm very new to XR and all that, but I assume that it's now in that's the free space right now, 
like we are in that era, that there's no too much commercial incentive, there's no too much 
like constraints around that medium that that helps people that are working in the, in this 
area, like just spark more ideas and, and better solutions. Right. So um, and like tied up with 
what we're trying to do. One of the things that I really like about this project decision is the 
choice that we're trying to bring back people's more choice to do what they really want to do, 
and not tied up with all these different systems that you need to comply in order to share, in 
order to do whatever you want to do. So it's very interesting. So he was not a question, just a 


remark. 
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Frode Hegland: You want to say more about. How this approach can be used. 


Frode Hegland: What is the thing that we didn't get to say that you have some more time. 
What do you really want to wait? 


Leon Van Kammen: I want everybody. Oh, now it already sounds a bit too much, but like, I 
think, like, forget about everything I'm working on. I think if everybody developers, game 
developers, web developers, if they just realize that every entity in A3D document is an 
opportunity to create a unique URL, so it's addressable by default. So you don't have to, you 
know, sort of come up with a URL. No, it's like if you just realize that everything inside A3D 
document has a URL already, then that is a great basis of sort of transcoding and referencing, 
addressing objects in in various various ways. Just a small example. If you have A3D object 
with a sky, you know, a sky clouds and you are already in A3D document, then it's really easy 
to just reference this document one child below the sky, because everything is hierarchical. 
The sky is the sort of outer parent entity, and you only have nested three d entities in them. So 
if you just if you're able to reference that document one level below the sky, then you can 
include three D documents into three D documents without having multiple skies or 
something because you only need one. I hope that makes sense, but that's a sort of revelation 


I got when when looking through the lens of 3D objects. 


Frode Hegland: Yeah, that is so perfect as an example. So you and I watched a bit of 
Hamilton yesterday. We watched it the night before. And if you don't know Hamilton, what 
are you doing in this room? Anyway? There's a great beginning, one of the first, when Aaron 
Burr is really mad at Hamilton because he keeps rising through the ranks. One of the lyrics is 
he changes the game. Right. And when you're a game designer in a computer game, your 
gameplay is called the meta, right? So when people are talking about the meta of gaming, 
that's what they're talking about. Of course we're talking about meta data here. And that's 
something that just people instantly generally just turn off. But metadata is the data that 
makes the data know what the data is. So if we change the meta data, we are the ones 


changing the game. 


Speaker2: I'm just trying. 


Frode Hegland: To paraphrase what you're saying. If it is known in an open system where 


things are and what they are, stuff can happen. It's like we're going from single celled 
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organisms of the web into eukaryotes who can spontaneously combine. So it's it's a beautiful 


illustration of what can happen. 


Leon Van Kammen: Exactly. And small note also metadata in the sense that it's already in 
there. You you don't need to to add it yourself. I think a lot of people, when they hear the 
word metadata, they think like, oh, I have work to do or I have to do something. But these 
days we have such rich file formats that there is already so much structure and data in there 


that we don't have to worry about that anymore. 


Frode Hegland: And actually, do you want to start very quickly? 


Speaker2: Sorry, we still have to use the special chair. The special chair. Very. 


Mark Anderson: Very quick one to pick up on the last. Sorry, it's Mare Anderson just thinking 
about this fact. Actually, one of the things I'm just looking at in a sort of large academic 
funding, basically the infrastructure for data, is that actually there is a problem of metadata 
because the metadata we need is the very, the very metadata the system can't provide for us. 
So we can tell us what that object is. It doesn't tell us why we made the object or what our 
thoughts or purposes. And often that's the very thing. It's the very thing we want to ask 
questions about and is the nexus of, for instance, some of why some of the current toys aren't 
as great as we thought they'd be. We can talk to them, but they can't tell us anything 
meaningful because all they ever see is the finished process. Because we haven't exposed our 
thinking and our construct through metadata. It's not an easy problem. I don't claim to have 
an answer, but I now think in terms of meaningful metadata as opposed to metadata, because 
it's metadata all the way down. It's another problem. So you pass through several levels of it. 
But I mean, there are some things that we don't capture at the moment and is a challenge for 
our making of tools, which is something that deals explicitly with the problem you 
mentioned, where somebody says, so I now have to describe this and they turn off. So a 
challenge for us all is to find ways to capture that I think is talk earlier. Yes. Because when 
we use new tools, new opportunities, we have to let some things behind. But one of the things 
we're going to have to grapple with is I don't think at the moment our tools can tell us why we 


did things, and that's one of the very things we want to know. 


Leon Van Kammen: This is a good point. 
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Dave Millard: Yeah, it was just a quick comment about Addressability. And so I think you 
have to be a little bit careful because there's a there's one of the things that people used to 
worry about was how do you address the destination or the source of, of relationships and 
links? And I'm trying to remember what it was called, but people used to talk about location 
specifications as a thing. And the point is that that that isn't just a piece of metadata. It could 
be a process. So you may not want to say it's, you know, I want I may want to link to 
everything that is blue, right? Or I might want to have something more complex. So so I was 
in Rome for hypertext. So when I came back IJ naturally watched the film Gladiator. So I 
might want to say, here's a link to Russell Crowe in Gladiator. And what I mean by that is 
every occurrence, every time Russell Crowe is in that, in a scene in which is most of it, but, 
you know, is in a scene in Gladiator. But that's a computational process. It's not it's not 
something that's already there. And I need to be able to have a way of kind of linking to that 
process that then dynamically resolves to something. Right? Or you might even think about 
3D spaces that change, like virtual world spaces. So every time I follow that link, I may get a 
different resolution because there's, you know, the space has changed or people have 
changed. So there are, there are there are places in here where we may want to have processes 
rather than rather than fixed specifications. I suppose that's. 


Speaker2: Why it's perfect because URL can be a query. Yes. Yep. Right. 


Leon Van Kammen: Exactly. 


Dave Millard: Well, you don't get them for Free is the point. 


Leon Van Kammen: I will. I would gladly send you the the spec because I think you would. 
You would really have some good comments on it. Or maybe I will cry afterwards, I don't 
know, but. 


Speaker2: All right. And hi. 


Keeichi Matsuda: I'm Eric, I'm on the mentor team. And my question is what we're talking 
about. Addressability in a in a space you imply that there's like a hierarchy that's like built 
into the 3D world. Like how the sky. So like, I feel like that's part of some unspoken schema 
that is actually specific to the document that you're talking about, the 3D document that 


you're talking about. And to use a quick thought experiment, let's talk about the 3D document 
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that we are all sharing right now. Right. Like how would you address in an absolute way 
something like, like a physical object that's in this room? Like is there a strategy that doesn't 


rely on some pre-baked schema that's already on the file? 


Leon Van Kammen: Yes, that's a good question. I found actually in the spec I've created, 
there's actually two ways. First of all, you have to realize that I found out that every 3D 
object has objects in them with a name already. There is a name, sometimes it's computer 


generated, sometimes it's manually, like this could be named table or it could be name x one. 


Keeichi Matsuda: We've named all of the elements that we have in our universe. Yeah, yeah. 


Leon Van Kammen: So so what you could do is like, you could just sort of reference, um. 


Speaker2: If you. 


Leon Van Kammen: Yeah. You could sort of reference this table. You could you sort of 
include the link to this table which is hash table. And then you would get everything, 
everything about the table and what is inside the table. You could even sort of like teleport 
yourself in between this space. I don't know why you would, but maybe you're like, no, I also 
want more. So you could you could also if it's if it's well designed hierarchically, you could 
just sort of reference the walls and then you're at least sure that you have the table and. If 
there are some objects in there which you don't like, you're like, no, that's way too much. 
There is always you can add query a query in the URL. You can say minus chair, minus this 
person. You could exclude some stuff or if, well, it's a bit of a mix. Like if you want to query 
more efficiently, we could sort of tag you. Just like in HTML you have class or and with 
stylesheet you can sort of it's a shorthand. Yeah sure. And you could sort of exclude all the 


persons in the room, all the chairs. And you could sort of or you could. 


Keeichi Matsuda: Some schema that like tells like the, the reader that, you know, I'm a person 
because maybe they don't understand and I'm just, I'm just matter in the room. But anyways, 
that takes the metaphor far outside of the actual 3D file. 


Leon Van Kammen: But the great thing I have to admit, I'm sort of a Frode Frode has inspired 
me to sort of stay away from becoming too specific, which I also started to look at BibTeX 
versus, for example, RDF. And I started to think like, well, well, if the HTML is the simple 
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version, which sort of, you know, was good enough for us, then I started to realize, like this 
BibTeX, which is just just saying this points to that, like this kind of sort of non-specific 
thinking. I kind of like it was also a great escape hatch for me to to not worry too much about 
what it really is. So it's more like a sort of like semantical soup, and it's sort of like the person 
has to make some sense out of it himself. So I do have to agree that is to make really sense 


out of it. That is really tough. That is really tough. 


Frode Hegland: So they're coming in to set up for lunch any second now or minute. So many 
years ago I got a phone call from Martin King, who was the guy who invented T9 texting. Do 
you remember that on the old physical phones you were predictive texting, and he asked me 
to take any book off my bookshelf, open to any page, and read one sentence, and then he told 
me the name of the book. That's used Google Books for that search. And his point was that 
English language is complicated enough that any sentence other than the most basic hello, 
and something people wouldn't never want to say, they are unique enough to where the data 
becomes the address. Right. And that I think you're referring to that. And that is so amazing. 
So if you're talking about a human being in that example, it depends who wants to know. 
Because if it's for another human being, then the attributes that make you you are relevant. If 
it's a clothing manufacturer, they're only going to care about the shirt literally. Right? So by 
describing the elements and making the elements, the way you get to it is similar to what 
Mark and I was talking with Chris Guttridge a lot TED Southampton, which is Addressability 
shouldn't be one thing, you know, for you to give someone else the Groucho Club. You said 
49 Dean Street. You can say exit the tube, take a left and a right. You know, you can ask 
someone on the way, which is why the notion of process is so important. So it's the same with 
locating something. And if you get to a point where the system isn't sure, that's okay, as long 
as you know the level of fuzziness that you are addressing with it, you can either interact with 
it someway or not, but just think the data becoming the address is just fantastic and any other 


things. Because I'm about to say something for the break for lunch. 


Anne Allen: So this is category. This is great data. What about the feedback. So I'm very 
intentionally curious about like what about sensitivity. Because there's so much that you can 
say without salmsons. But that's so you lose a lot if this is where we're going. How do we not 
throw out the baby with the bathwater? By stopping? 


Frode Hegland: No you don't. Because first of all, metadata can be inherent and it can be 
administrative and descriptive. It can be all kinds of data. Right. But if you look at Nayana's 


presentation on the the orality, you know, what is the metadata there? Is it the room it was in. 


419 


Was it the is it the time? Is it the intonation? The intonation is really important too. But also 
like when we watched him last night, he writes a letter to Angelika, who was his sister in law, 
and he adds a comma between my dearest and Angelika, right, changes the meaning. That is 
subtext, right? I mean, what a system would necessarily care about that, right? So you can 
still put it in. And also I do think that voice will come back a lot more. Olivia, I agree with 
you a lot more. I also think that handwriting will really have a resurgence now that systems 
can interpret that much better. So we will have to choose what we want to have in 


generations. 


Dave Millard: So I was just going to say, I think it's worth making a distinction between 
metadata that is in the data, and you'll just pull it out so it's machine readable and metadata 
that isn't in the data. And I think you're talking about the problem of saying if you're pulling 
out metadata that's in the data, what about the stuff that's there that you haven't pulled out and 
you ignored? And of course, it is still there in the data you're only doing. You're only doing 
that to for an engineering reason. So you can. Yeah. You know, I think it's worth thinking 


about it in those different because it goes. 


Mark Anderson: Back to what you capture or what you choose not to. 


Speaker2: Capture. Yeah. 


Frode Hegland: And what's inherent. So expecting them to come barging through the door. So 
this is a piece of obsidian not the soft software. The very very very hardware. And I bought 
this this summer in Syracuse, volcanic Sicily and all of that stuff because it fits so incredibly 
well in my hand. And that was really, really odd. I'm going I'm going to toss it around like 
we're in a school class or something. So the point of it is the folds of it, the way it naturally 
cuts. As I say, it fits this so well. And so I was wondering about that. We've been using that 
for longer than we've been humans by basically an order of magnitude, you know, like more 
than a million years rather than humans, what, 200,000 years? So then there is the research 
that shows that maybe our tools shaped our hands, which is kind of neat, right? Because we 
were Homo habilis before we were Homo erectus. We started using tools before we were 
standing up. And the reason I want to hammer that in your heads is because the tools we 
invent now will shape the minds of every human after us. I just cannot bang the importance 
of the technicality. Who cares? It'll be higher resolution faster. But the first steps in what we 
choose to do in this space. It's the last time in history we aren't using it. In the future, no one 


will have the benefit of our morality. When you talk about orality, you were also reflecting on 
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morality. For people who don't know how to read and write, being different from speech by 
someone who doesn't need to know how the brain, you know, in 1020 years, no one will have 
the benefit of not knowing what a headset is. 


Dave Millard: And look at the influence of HTML on this conversation. Technology. Getting 


around that is tremendously difficult. 


Anne Allen: Are you saying no one will have money? And what does that mean? That's good. 
It doesn't you. 


Speaker2: It's. 


Frode Hegland: I think it's inevitable. More than anything, I'm more of a light McLuhan ite in 
the sense that technologies change how we think. But, you know, a level of how deeply and 
how much is. It's just it's going to happen. We might as well keep it in mind. You know, a lot 
of technologies, you know, opportunity when it goes up will equally go down. You know, the 
opportunity for crowds, you know. So we have to take more ownership of it. Ownership, 


ownership ownership. 
Anne Allen: So the big question about what sort of future society we decided to put in place. 


Frode Hegland: And that's why we need to have a bigger discussion of who should be in this 
community and communities like it. Doug Engelbart had this notion of an improvement 
community, that there will be many and there would be networked improvement communities 
where all their parts with lots of different things, just try to make the overlap and the dialogue 
more. So yeah, very provocative ending. So yeah, let's break for lunch and thank you 
everyone. So online people, we will be back in an hour but we'll leave one camera going. 


Thank you everyone. Thank you. 
Transcript: Livia Polanyi 


Livia Polanyi: So my name is Livia Polanyi, and oddly enough, many years ago I wrote a 
book called Telling the American Story, which is actually about the constraints the linguistic, 


social, interactional, and cultural constraints on oral storytelling, conversational storytelling. 
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And so I think it's a very there are some very interesting issues when one thinks about how 
how things worked, how things work with oral storytelling, with actual people who are 
telling actual stories as the tellers versus what happens when in fact, all of us tell stories all 
the time in conversation about what happens. But that's so that was just a very interesting 
kind of thing. And the whole question of orality will actually enter into this as well. So this is 
a short note. It says on writing letters and letters in this in this presentation, I was basically 
refers both to actual letters or characters or. Oh, syllabaries or whatever, which people use 
essentially to put one after another to make a word and then put those words one after 
another. And it also refers to the idea of actual letters that people write. So starting off, you 
know, we had cuneiform, which obviously we had the technology involved. There was quite, 


quite elaborate and very much restricted to a very few people. 


Livia Polanyi: When we moved on to. Uh, working with letters that were sent to that people 
sent in documents which were on paper or parchment or whatever. But when we think about 
people actually sending letters to each other, well, there was this technology, that of the 
scribe. You would go to a scribe, and that was the technology you used actually, to write 
letters to people. And that persisted. In fact, it persists to this day with people who are 
illiterate. But in general, what has happened is that literacy has passed on. And and so the 
kind of. Technology that was involved. There was really the the knowledge of knowing how 
to write, but also the development of lightweight pens, paper, that sort of thing. That in the 
middle of the sort of towards the end of the 19th century, of course, we had the invention of 
the of the typewriter and the use of the typewriter. And then also at that time, the 
development of the dictaphone at the beginning of the very, very beginning of the 20th 
century. And what's happened then is where we are now, essentially, which is we have done 


away with a lot of technology. When it comes, for example, to writing letters. 


Livia Polanyi: Okay, people don't buy stamps anymore. Okay. So what we do is we sit there 
and print them out, or most likely use email. So then the question is sort of what's going on in 
the future. So this actually started with my thinking about, well, when was the last time I 
actually got a real letter, and when was the time I actually wrote a real letter. And you know. 
No. We get all this stuff in the mail. Okay. Marks on paper, as it's called. It was called a 
Xerox. But when was the last time you got a letter that was actually people telling you about 
stuff that wasn't like a birth announcement or, you know, a legal document or something. So I 
was thinking, you know, about that. And so what's happening now, of course, is that all that 
stuff that we did internationally, a lot of that now we're doing on text. And that's true kind of 


all over the world that people are doing it, texting each other the kinds of things. But still 
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what the way that's working. Right? Letters in one sense, but one letter character at a time, 
basically to get the message across. And of course, what we have with kids is that they're all 
again all over the world. Children are basically growing up in entirely digital environments 
where text, email, that kind of thing is. And we're not even talking about obviously, images 
and things of that sort. But this at the moment of talking about letters. So what's going on 
then? So as we move then into this idea of the future and particularly in, in virtual reality. So 
one of the things that I found really amazing is that there are all I mean, I'm not a VR person 
at all. So finding out that there were large numbers of proposals for keyboards and how you 
deal with the keyboard to get this one letter at a time thing that's going to turn into a text. And 
I kind of wonder, how much sense does that actually make, really. Now, adopting new 
technologies has always involved abandoning some kind of perceived affordances of old 
technologies. Right? So basically we, you know. You got a lot of stuff out of that scribe. 
Okay, first of all, you did the work. Note spelling problems you didn't have to have. You 
didn't have to provide the the paper and so on. So. We always have to give up affordances 
from one situation where we go into an entirely new technology and. And so in a world 
essentially of increasing conversational AI and oral interfaces, it really seems to me to be a 
little silly to be divided. I mean, I've been amazed. I mean, if you type in, you know, 
keyboard. The keyboards in in VR. I mean, really odd. 


Livia Polanyi: So it's a wonder. No problem. So, you know, I'm a I'm a linguist. I've spent a 
lot of time in people actually think I'm a computational linguist, which is kind of hilarious. 
But I've lived my a lot of my life in the NLP community. And so really, I'm back to orality on 
this one. I think that there's going to be a lot of use for speech, speech, speech input. Through 
speech recognition, but then of course, also natural language processing, because one of the 
things actually that I found when I was at Microsoft talking to the top, top people in speech 
recognition. I was very surprised that they didn't actually seem to understand that the problem 
of was not only recognizing speech, because if the problem was actually getting speech to 
text. That was the only problem that would have been, you know, we would have been 
talking to computers and getting our information out of nothing for years. So actually the 
problem obviously is understanding the text. We can then imagine understanding what has 
been processed. So we're not doing this anymore. We're doing what we normally do. And. So 
the idea people are still going to write letters to tuk. Here are a few things. And. But my point 
is composing texts by entering one letter at a time in virtual reality, I just don't think is going 
to be the way it's going to work. Dictating. Reading. Editing? Absolutely. But. One letter at a 
time in order to compose your texts, I think is. 


423 


Nayana Prakash: Thank you so much. That was really interesting. And as you said a really 
good follow on from like the previous one as well. Um, so I'm going to ask you maybe kind 
of the same question someone asked me, which was about dialogue. I was thinking about 
how Walter Jong talks about how writing is a technology that restructures thought and every 
kind of technology we have kind of maybe frames the way we think about conversation, like 
letter writing is you write a whole bunch, and then maybe a couple of weeks later you get a 
response and then text is is relatively immediate. You know, you write a couple of sentences 
and you get a response quickly. So how do you feel like this kind of future of text is 


restructuring dialogue and conversation? 


Livia Polanyi: Well, one of the actually one of the interesting things about. If you will look. 
Computationally modern mediated interaction is even things with texts and whatever. Is that 
there is async, an asynchronous aspect to it. And when there's an asynchronous. Aspect that 
does actually some very strange things. I mean, one of the things it does. Is it changes the 
meaning, for example, of words like now and. And certain kinds of things. Like, for example, 
the reference of pronouns can get screwed up, which is which actually does not happen. You 
know, with almost no exceptions in ordinary conversation. That's been the last, you know, 
after oral story, after conversational storytelling, I actually got into questions like what's 
going on in now? And then ended up building a essentially formal model of the structure of 
discourse, which sort of accounts for a lot of this stuff. So. Yeah. I mean. The thing about. 
The thing about textual representation of speech, okay, sorry, is that there's actually a lot 
which is not going on as people text them. And one of the things that I think is very 
interesting, and we all sort of noticed it is when is when we are in, let's call it a conversation, 
a text conversation. You know, you've happily, you know, put in your text and then you see 
that this other person is and then suddenly it disappears, right. So there's possibilities, for 
example, of correction, which is not a possibility in a purely, if you will, oral interactive 
environment. Yeah. And and so and I think that whole business is, is exceedingly interesting 
and actually relates to the point that you made about flattening. Okay. Because there's a lot 
that goes on in normal interaction which cannot be flattened after the, you know, even 
though. No, I didn't mean to say that, I didn't mean that Freudian slip and so on. Anyhow, the 


that I think there's a long conversation we can have. Yeah. Perhaps. 


Nayana Prakash: Definitely. Yeah. 


Fabien Benetou: But I'm going to argue it's a wrong step because it works. It's terrible, but it 


works. And sadly, now that it works, I'm like, well, I mean, I could do better, but I'll have to 
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actually think and it's hard. So. So I really recommend if you want to explore this like any 
other solution, micro gesture or speech or whatnot. Don't have a fallback with a keyboard like 


a bad one with an a physical one also, because then you fall back to it and. 


Fabien Benetou: So that was the one I'm. Fabian. Fabian. Vinny to build prototypes at the 
European Parliament, especially on on XR for knowledge management because I'm pretty 
sure that's. Yeah that's one of the most interesting, exciting interface that works today since I 
wrote in my backpack a couple of hundred bucks headset and it's just there. So but yeah, that 
being said, how to not just type also and input but edit actually after it's. Yeah, it's it becomes 
one of the most interesting question because it's one of the most interesting interface. But I 


don't have an answer. 


Livia Polanyi: I think that the how shall I say? There are affordances, okay, for editing where 
keyboards are may very well be useful. So the real question here, which I find is thinking 
about. You know not. That you are going to lose certain kinds of affordances when you 
introduce a new technology. That you shouldn't necessarily impose old solutions with their 
affordances. But what that is going to mean is that you're going to have to innovate to be able 


to capture. New affordances to to do jobs, which you were doing in using the old technology. 


Frode Hegland: So I say, I see Fabien in there, but I'm actually going to jump in a little bit 
before and also after Libya. It's going to be since you didn't get to do his opening statement, 
he's going to do his leaving statement because he has to go next. And also, if you're 
presenting online, you'll be presenting after lunch. We want to make sure we can hear you, 
which would be useful. I just want to thank both of the last two speakers, because if this was 
a year ago and you came in here and talked about voice, I would be really stressed and angry. 
But what is happening now is much more interesting. I always thought of text as being 
something that can be forced multiplied by other media, but that was primarily images. You 
put a word in a picture, you know that's a bit obvious. So what I'm hearing about the different 
kinds of morality is the morality where you are illiterate, of course, completely different. You 
don't even space the words the same. Right. And what you're talking about here, going into 
this new space, I'm really I'm so grateful for this dialogue because we need to just like we're 
talking about the definition of hypertext or hypermedia. We need to shed all those stupid 
terms and really look at when is it actually best to do this and how can it be combined. So if 
you have any further comments, I'm happy. But it was mostly a gratitude for this dimension 


to the dialogue today. 
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Livia Polanyi: Let's just to make one other comment. I'm a totally text person myself, and so 
it was only because the topic here was virtual reality that I began actually to be thinking 


about this particular issue of essentially text input. 


Frode Hegland: Yeah. Thank you. Yeah. Thank. Oh, yeah. Brandel. Sorry I will I think I may 


have even muted you please come back. 


Brandel Zachernuk: Yeah. Thank you. Your reminder of the function of the scribes in a sort of 
an intermediate oral tradition, just set my mind afire. With what? The sort of covert function 
of a scribe was, which was also to help communicate and craft a message as well as simply 
be responsible for the transcription. And it made me think about the scribal traditions that we 
both may have in a future environment where people are able to write, but they're not 
necessarily able to perform or communicate a message as eloquently within sort of a modern, 
pervasive hypermedia. But also we have scribal folks who do things like diagram or illustrate 
already. So there's there's already a rich contemporary tradition where we are interposing 
these, these scribal functions. So I'm very excited by that. But I'm wondering what other 
points of reference you would draw on in order to reimagine that sort of intermediating 
function within a context where nominally we all have the same capabilities to use these 


things, but some of us are better at making those messages happen than others. 


Livia Polanyi: Yeah, well, one of the things that in my presentation, of course, went by very 
fast, but there was this little thing of ChatGPT. Right. And, and essentially the danger of. Of 
that kind of technology. For being seen as some kind of digital. Digital scribal knowledge 
source. Okay. So when you were actually there with your scribe and, and if you're lucky and 
have a really good patent attorney, they will also ask you the right question about what you 
didn't. You mean. And it was like, oh yes, I did. And yeah. So there's, you know, that other 
person. Yes. You would perfectly well say, gee, I am writing, you know, to my mother in law 
and she doesn't like me, should I, you know, what word should we use or. Yeah. So that scribe 
had a lot of had a lot had a lot was part of the technology base of that scribe was to have this 
knowledge that they could, that they could give to you. And of course, we have, we have. A 
very important scribe, and I don't know about all of the the legal systems outside of the 
United States, but in courtrooms in the United States, essentially the document of record is a. 


Is a transcribed is transcribed by a person. 


Transcript: Fabien Benetou 
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So. Hello everyone. Here you can hear me. And online. 


Keeichi Matsuda: So perfect. Super. Thanks. 


Fabien Benetou: So I am Fabien Benton to I do Web XR _ prototyping. That means virtual and 
augmented reality on the web mostly. I mostly do it for the European Parliament and for 
knowledge management because I want to have more ideas. And one simple way I manage is 
to write them down. But I feel this tiny screen, those two millimeters of plastic, it's one of the 
worst way to do it. Uh, so yeah, I started to do other things, but those are my slides so that 
you will not see them. I will need to share. Just click. 


Frode Hegland: And I've got to pin you. There we go. And get rid of everyone else in the 
world. 


Keeichi Matsuda: Test this. It's not gonna read it, though. Well, I'll also talk, and I'm working. 


Fabien Benetou: On my handwriting, among other things. 


Keeichi Matsuda: Cursor left. Cursor left. All right. Okay. 


Fabien Benetou: New technology. Not that easy. Okay. So. Yes. Last year, for the future of 
text, I also made a short presentation at five minutes, but a bit longer on interfaces because I 
think basically everything around us is an interface. We mentioned briefly affordances, but 
affordances toward information digitally or just the handle of a door basically. And you have 
API how you're going to manipulate information in software. So basically my conclusion was 
everything is an interface, which I think is true. The question then is is it actually useful to 
say it to say, for example, that text is an interface, that having a discussion is an interface to a 
concept. And. As I mentioned briefly, my life, my professional life. But personal life is being 
a prototype. So I want to build stuff. And so I have to be a bit more pragmatic. What is an 
interface? Do I need to build an interface? Do I need to build with an interface? And we 
discussed just over lunch about Lego. Lego being an interface to a physical world, to being 
able to build an actual thing for games or more. And also noticed that what makes me happier 
is not just to talk about things. It does help. But intellectual masturbation can only bring you 


so far. At some point you need to make something. So I noticed that what makes me happier 
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is yes, do discover something new, but then build something out of it and something that has 
an impact on my life or the life of people around me. So basically also as yeah, not just 
theoretical fact, but build the thing. The next slide. One of the very simple things I built 
recently. To give a more practical example, a lot of you, I guess, are familiar with screen 


time. 


Fabien Benetou: How are you going to limit your own behavior? Like, oh, I spend too much 
time on whatever platform I want to spend less time on it. So that works on my phone. But I 
spend a lot of time on this on the laptop, on a device like this. So I made my own equivalent, 
let's say, of screen time, where if I spend too long, let's say chatting with some people on a 
certain platform, it redirects me to a better alternative. So basically, well, yeah, whatever I 
find better actually, whatever I when I'm at maybe not peak of my capacity of wisdom, but 
something that I'll find better, which is basically just rerouting. But I'm in a way editing my 
behavior. I make an edit of who I should be. I do it through text, through code, because that's 
what I do. I do it also tangibly, because I started rollerblading six months ago, more or less, 
and I somehow broke them. But I have a 3D printer at home, so I fix. I edited my movement 
through modification of code to generate a 3D model. So I'm basically editing again, and it 
makes me happier by controlling my behavior, my movement. So basically I thought, well, 
editing seems to be a big deal. So next slide, which basically a bit like last year something for 
everything. I want an edit button for everything. Literally. That's the whole idea. I want an 
edit button, of course, for whatever is on my computer or on my phone, whatever computer 


actually might be. So including also if you can hand me the VR headset there, please. 


Fabien Benetou: Thanks a lot. So I made an edit button during lunch, so it's still not a great 
one, but it is there. So I have my edit button and I think it's actually feasible. It sounds, I 
guess, like a silly theoretical idea to have an edit button for everything, but I think we are in 
interesting intersection where we're running Linux a bit everywhere. We have browsers that 
are open source, we have open source hardware. I have my backpack somewhere I'll show 
you after, but I have some like augmented reality lenses to clip on glasses. It's all open 
source. So we have the software and the hardware stack was mentioning 3D printing. So 
digital manufacturing. So I think this and identifier like Doi, ISBN, URLs also that were 
mentioned before make all this actually feasible. I think if I say I want to edit that table, and I 
have the model of that table and workshop on soldering a couple of months ago, then I can do 
it if I want something with this, another tool, and I know how it behaves because there are 
resources online for this, then I can also edit it. Is it combining them? Is it combining the 
three of them? I have absolutely no idea. But I think the the idea of it and actually making it 


428 


is feasible. One little thing also is that's why I mentioned or show the headset before is it has 
to be done right here, right now. It doesn't have to be done through this laptop for that thing, 
for that thing. 


Fabien Benetou: It's like if you have a device you need to edit, not the content of it, of course, 
the content you have watching in it, but the device itself and you do it through that device. So 
I call it responsive programming, not responsive design, but you change the device through 
the device itself. So yeah digital non digital. Being able to inspect different layers from let's 
say physics chemistry whatever you want with headset, no headset. You can edit the edit 
button itself. You can have an edit button on yourself. Because maybe that's also in term of 
agency. And what brings you happiness. One of the biggest impact. And you could do it 
socially. For example, if I edit this table and I put metadata online on how to do it, others can 
benefit too. It's not like a individual behavior. It's something that can be shared. And as a call 
to action, I actually have it's a beta version, but I have a couple of such edit buttons that I'm 
going to put on this table, and I'm going to hint to people in the audience they're not good. 
They don't do actually anything right now because it's a couple of days old idea. But I think 
the prospect of being able to edit literally anything and actually do it, like implementing 


what's behind the button, is a worthwhile and exciting endeavor. Thank you. 


Frode Hegland: We in the question chair. 


Keeichi Matsuda: So I need to ask. Right. Okay. Well yeah. 


Frode Hegland: That's the that's the thing. 


Gabo Beaumont : Uh, that's what I'm here. I'm it's very interesting. Like, this reminded me 
this, um, there's a new trend in, like, I'm all into text editing, especially in the browser. And 
there's a new trend called programmable media, which is exactly like generating, like, the 
same things that you're seeing or you're editing are capable of editing the whole system. Or 
you can add variables in a system in that system that you can reuse in other places and create 
more complex scenarios. Right. And this reminds me a lot like the fact that you can have one 
building block that can be as complex as you want and create more elaborate solutions. So it's 


kind of interesting idea that very. 


Fabien Benetou: Little if I jump on the back of this is the system like this is a system also 
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where this place is a system also. So we tend to especially as a developer, just think about the 
digital and software behind it. But I really think again like it's a silly example. But the fix for 
my rollerblades, yeah that just work. And it's a physical thing. All the workshops we, we, we 
can do about, I don't know, uh, many working woodworking or this kind of things like it's 

let's not stuck it behind a screen in a digital way. It's the intersection of both. I think that's the 


most impactful. 


Keeichi Matsuda: But. Well, do you have any other examples? Of things that themselves. So 
did you repeat. 


Frode Hegland: The question, please? 


Fabien Benetou: Yes. Do I have other examples of things that edit themselves? So one of the 
things I'm working on now on VR, I call it like a self editing system, but I think that's a tricky 
shortcut because then we we start to think it's a magical solution. It's more yeah, it's literally I 
want to label the or stick a. I did not have the time for today. That's why I have this. But but I 
want to stick an edit button on literally everything so that I get the habit of it's not. I'm here to 
consume or use. I can, of course, if I want to use that table as is, but it's just like a step. It's 
version one or version whatever, but it's just a version among others, so that they don't 
necessarily edit themselves. As much as we mistake the habit of saying, okay, this is the 
current version, I can more easily now access ways to see how it's done. See, I can change it, 
learn if I don't know yet how to change it and I don't know, polish it or whatnot. So yeah, it's 
more building the interface and the habit of allowing ourself and having being confident 
enough that we can edit in terms of actual implementation. Like I said, because the idea of 
couple of days old, I don't have, let's say, a repository and the things that I, I'm being a bit 
naive, but I guess that the easy part to start. 


Keeichi Matsuda: Your job as. 


Frode Hegland: Part to be naive, that's the whole benefit, right? So. 


Keeichi Matsuda: Well, when. 


Frode Hegland: Talking about examples, just, you know, we're both interested in children 


growing up and how they see the world. It could be interesting to just make some stickers just 
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like you did, and put it on notebooks. And a notebook says, this notebook is editable. 


Keeichi Matsuda: Yes. Yeah. 


Fabien Benetou: So this sounds a little bit silly or it seems a bit silly, but it exists. So my job 
is being a prototype as a prototype is to make prototypes. So that's a I would say it's not a 
terrible prototype again, because it exists and I can literally put it there. And the function let's 


say has begun. It's far from enough, let's say. But there the buttons to edit anything is here. 


Keeichi Matsuda: But that's what's so important. 


Frode Hegland: You know, we're talking about dreaming in this community, and you dream in 
a very specific way. You dream beyond just writing down your idea, you know, by tearing a 
piece of paper and writing something on it that is a substantiating of that thoughts. I think 


that's very, very important question. Oh, sorry. 


Keeichi Matsuda: What? 


Gabo Beaumont : Oh, in the chat there was oh. 


Frode Hegland: Oh other person question. 


Keeichi Matsuda: Oh. Scalicus. Okay. 


Fabien Benetou: So I'm going to read it. I guess you prefer I read it aloud or you prefer to 


read it on? 


Keeichi Matsuda: Okay. So I guess. 


Fabien Benetou: I need to read it first. So, um, just we did that presentation with a 
placeholder about how this naive ID feasibility. Okay. It's not good if I read it too fast, it 
doesn't help. 


Keeichi Matsuda: Not any one of us. Okay. 
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Frode Hegland: Can you make sure that laptop has power? 


Keeichi Matsuda: Yeah. So. 


Fabien Benetou: Um. Yes. It's, um, I don't know. I don't want to dig into mineralogy now, but. 
Yeah. What's the thing when you put an edit button on a thing, what is that thing that you can 
going to edit? Is it the table itself? Is it I don't know the. The process that led to that table that 
you're going to edit. Are you going to edit, let's say just the glass or whatnot, as I think the 
being able to define or that's why I mentioned also briefly layers in the presentation. Are you 
going to work at the physics level, the chemistry level, the social level, etcetera? I think it's 
indeed that interconnection of the different layers that make it complex but also beautiful, 


because then you can compose, then you can have a lot of different ways to do it. Too late. 


Keeichi Matsuda: Yeah. I think. Yeah, I think if we're dealing with media that is intrinsically 
abundant, then there is no issue with anybody going in and making their edits and having 
their version that is personal to them. But as soon as we move into scarce objects, which is 
anything in the physical world and many things that are shared in the virtual world as well, 
then this idea of anybody being able to edit becomes instantly a very political. Yes. Political 
issue. Who is allowed to edit? What are the permissions? How do you grant those 
permissions if there are multiple versions of a thing? How do we understand which version 
you're seeing and which version I'm seeing? I'm very curious in terms of like how far you've 


run this thought experiment and if you've run up against any issues like that. 


Fabien Benetou: Um, so again, very fresh. So I did not go in depth there, but it's okay. It's 
going to sound a little bit pompous or grandiose, but the point is to be revolutionary. The 
point is to say I can just be a consumer, but at least individually, personally, that doesn't make 
me sufficiently happy when I have agency against a thing. And I just admit that I don't 
actually know how to change it, even less improve it. But if I can dare try, that's that will be 
an interesting process. But it is revolutionary. It's a kind of, let's say, challenge of power to 
say, oh, I can't edit it how it's going to, not just between objects but between people. Maybe 
you love this table the way just like that, just the way it is. So that's for us together to figure 
out. So but but that's yeah that's. 


Frode Hegland: To share ability and depth. There's also mathematics so I'm really bad at 
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maths. Sorry Dave you're over there really bad right. Trying to teach my son a few fractions 
and things in his head. But the reason I'm saying it is when I'm now trying to learn a bit of 
mathematics from the real brilliant people. They all keep saying mathematics is 
experimenting. There isn't things there you have to make postulates of. Mathematics itself is 
editable. Yes. Right. So in school we are taught this plus blah blah. You know that stuff to 
have the mindset you're talking about to take ownership of editability creativity and 


involvement is crucial. I mean, yeah, thank you. 


Fabien Benetou: One of the quick thing, I did not draw it. I don't think so. But you have a 
USB stick with a Lego. 


Frode Hegland: Whose was that. 


Fabien Benetou: Again? Who has the Lego USB stick? So there is a Lego USB stick. It's tiny 
so you won't see it. But. But what I love about it is that you have the Lego affordance, the 
interface to build more on it. Whereas if I take the exact a normal USB stick, let's say it's 
going to be a flat surface. So that's it. It's as it is. Now that you have those things to clip on, 
you have an interface to build more. And I think we used to have flat things. And I think if we 
could still have the flat thing, but can we move this to clip on? Let's say it's a metaphor for 


everything being math or something else. 


Keeichi Matsuda: Yeah. Thank you. Yeah. 


Frode Hegland: If you can summarize them. Because you're out of time, I'm afraid.. 


Fabien Benetou: Yes they are. So they were being graffiti and we I. So I have my own wiki. I 
have a network of wikis actually, so that there is a history of edits literally behind this I. A Lot 
of different examples. Graffiti also being quite. Yeah. In your face I guess obvious about it, 
but I think it has. It's not just revolutionary to be revolutionary or challenging literal 
structures in places. Yeah. Try to collaboratively, individually build something better. But 
yeah, it's not. Editing itself is not new, but I think having an actual button that's going to fight 
for it and implement it a bit and just do a little bit, I can't help myself. But let's say tech or 
geeky promotions you think about, because we mentioned trendy topics at the beginning, 
editing the headset itself. But the headset, especially if you think about augmented reality, 
you look at the real world through the headset, and then if you display a button toward or on 
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top of objects and you have the headset to learn how the object functions, it becomes quite 
interesting because you can bootstrap the learning process to modify it a lot more than just 
the thing with a button on it, because it's a dynamic system like you can simulate how the 
object is going to behave, not just look at it as, I don't know, a 3D model or something static. 


But I think this also to do a little bit promotion for XOR is going to is not coincidental. 
Frode Hegland: Okay. So you have to say that sorry even if it's quick. Sorry guys. 


Leon Van Kammmen: And so thank you very much. Really interesting. I have one question. I 
was a bit. I got all kinds of ideas from you. Like suddenly Lego is a. I'm now seeing it as an 
editable system, basically. And perhaps when you were using the paper, like the pen and the 
paper is also a sort of added, added button. Um, then also like I started to think about this, uh, 
I don't know what it was called, the right to repair or something. Can you also like, is there 
any connection with your thinking in relationship to this right to repair? Yes. And where, 


where? Yeah. Could you maybe tell something more about that. 


Fabien Benetou: So we can piggyback on right to repair for right to edit? It's. Yeah, the right 
to repair is also like, this is my hardware. It's not a lease from, I don't know, a big company or 
something like this. It's something that I don't have to. Look at statically, but I can fix it. But I 
think the fixing it is still like a first step toward more not accepting that the design is finite. 
This is done, this is finished. It's finished. And for now that's the version that's been shipped. 
But I don't have to accept it as is. And I think indeed the right to repair and what what we get 
hopefully more and more like in France, for example, where you have to sell the device, but 
you will have to sell it with the interface to repair it. So the right set of tools, physical tools, 
then I think it's definitely going in the right direction. It's not enough, but and just I don't 
know if you can see it on the screen. Now the edit button has an edit button. So again it's not 


perfect but finite. 
Transcript: Keeichi Matsuda 


Keeichi Matsuda: Hi, everybody. Yeah, thanks. Frode. Frode and I go way back. We met 
yesterday, I think. Not really. Yesterday he he sent me a message on Twitter saying, do you 
want to come to this thing? I said, okay, so here I am. Very nice to meet you all. I'm a 
designer and film maker. I'm based in London, in East London, and I'm just going to show 


you, I guess, some projects which I think resonate a lot with what I've heard so far from this 
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event people are interested in and my background is in architecture, and I got very interested 
about the possibilities for space of being able to create new types of environment that were 
composites of physical and virtual. So I was very excited to find out about augmented reality, 
because it made me think that there is this kind of way of bridging my interest in physical 
spaces with people and face to face communications, but also being able to draw possibilities 
from the virtual world as well. So I've got kind of two themes I guess I want to talk about. 
The first one is another hyper X, and I didn't bring this this, I didn't take this, this, this phrase 
from from hypertext and hypermedia. It really comes from Baudrillard. But obviously it's 
very relevant to a lot of the work I'm doing. I'm just going to play some of my, my work, 
some of my films in the background. This is way back in 2009, when I started to think about 
what are the possibilities of physical and virtual together. 


Keeichi Matsuda: I'm sorry it's stuttering so much, and started to think about the possibility of 
media escaping from these 2D surfaces and starting to form environments. Information 
environments that could be navigable, could be interactive, could be very mutable. I've got 
the obligatory air keyboard in there as well. So 2009. So, you know, that was, you know, the 
first time I was thinking about any of this stuff. And you'll see I quickly move on from those 
ideas. But yeah, just this idea as well that, you know, we could layer on top of our space. We 
could actually even replace our physical environments with virtual ones and start to think 
about space not only as a place for us to exist in, but also for us to to consume as a form of 
media. I kept on with this kind of train of thought. I'm trying to think about who creates that 
media. You know, if it's a position where we have this edit button for everything and we're 
able to be able to have this fully plastic reality, what, you know, who is allowed to do things 
in the space. And I was very excited as an architecture student in that time, that it wouldn't 
just be architects and planners and developers who are responsible for defining our spaces, 
but it could actually be the people who exist in those spaces. They could leave their own 
trace, they could broadcast imagery about themselves into that space. And the fact that we're 
all here at this moment could be meaningful for the way that we perceive this environment as 
well, in the same way that everyone is putting their their slides in the in the screen, you know, 


there's a possibility for that information to be held in the very physical space that we're in. 


Keeichi Matsuda: I followed that up in 2016 with this film, Hyper Reality, which I guess sort 
of takes that to the next level and tries to imagine what that looks like in this kind of urbanist 
perspective, where, you know, people are responsible for being able to maintain and share 

that digital content. So here you can see that this is shot in Medellin, Colombia. And each of 


these little shops has their own virtual presence in the world. There are municipal layers, 
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there are fully personalized layers. And I was very interested in thinking about that app model 
where you have this scarce space that can be accessed and created by many different people, 
and, you know, how does that work? And of course, I think in these films you've seen so far, 
these are not commercially funded films. I was able to put a critical eye on those things and, 
you know, try and sort of just extrapolate. If we continue with the same business models and 
kind of extractive uses of technology, you know, what does that what's what's the kind of 
logical conclusion for that. So in a way, these are kind of provocations trying to make people 
think a little bit about what kind of future are they trying to create. 


Keeichi Matsuda: Why are people using these technologies? And yeah, this is a film from 
2017 looking more into virtual reality and artificial intelligence. And I was very interested 
that, you know, I started with one screen in front of me, and then I bought another screen and 
I had my laptop and then another screen. And I thought at some point, you know, we're 
actually going to dive into the screen and we're going to create this very pure information 
environment. So this is a film about the future of work. Actually, it's a 360 film that you can 
watch in the headset or pan around, but it it proposes many different types of these 
information environments, which, which can be for productivity or for understanding data 
sets. And you know, our character here is trying to understand this unknowable. I mean, these 
all have narrative. I'm just showing you little, little glimpses of them. And then we got the 
chance last year to be able to really build this for real. I mean, I was working in 2009, 2010 as 
a student to try and just visualize through computer graphics and just conventional, you 
know, CGI techniques what that could look like. But now, of course, these things are fully 
realizable. So in last year, we built this installation for Niantic, the makers of Pokemon Go, to 
overlay their entire conference venue with these virtual layers that you could browse and 


navigate. So I think here we've really tried to look into the utility of what happens. 


Keeichi Matsuda: If you could put the conference agenda up in the sky, you could be able to 
follow the thread of what conferences are now to get to the room you want. We can tag 
landmarks. We can tag what kind of the provenance of your tea and coffee and, you know, 
really trying to dig down into those use cases here. We don't touch so much on the extractive 
business model side of it. And yeah, I think that's really, you know, this preoccupation that 
I've been, I've had for for over a decade now is trying to understand what is that connection 
between the physical and virtual spaces and these information environments. But the second 
strand I wanted to talk about is agents, which is somewhat new. I started kind of thinking 
about this in 2016 and really just recently put out my latest film about that, but this is just an 


example of some of the prototyping work I was doing while I was at Leapmotion. And again, 
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here, this is really in the hyper reality mode of things, trying to think about how can we think 
about new forms of interaction modalities, what can we do to be able to to sort of imagine 
ways of being able to interface with the world around us? So it's not just consumption, but 
there's also a way of being able to modulate, control and contribute to those worlds as well. 
So these are kind of just tests, I guess, to show, like what happens if your your body becomes 


a form of, of interface and there's a lot more like this as well. 


Keeichi Matsuda: But I think the realization I had with this was a similar to the virtual 
keyboard case, which is that I don't necessarily want to be in this space, like looking at these, 
these things or like touching an air and, you know, maybe the the paradigm of buttons and 
text surfaces, although it has many usable applications, I don't know if I am fully convinced 
myself that that's the future of how we interact with computers and the space around us. So I 
started to become much more interested in the possibilities of conversational UI, and I made 
this film recently. It just came out a month ago, I think, called agents, which is a narrative 
film. Again, it's going back to the roots of just faking it as a as a film. But here we try to 
imagine that we have these artificial intelligence agents which coexist within your shared 
world and can have different functionalities. They can be made by different people, they can 
have different capabilities. Some can be very smart, some can be very stupid, some might be 
benevolent and fully aligned with your interests. Others might have the interests of a 
corporation or a brand or something else. So I was very interested, I guess, here about just 
what you can, how you could go away from that model where we have the kind of 


superintelligent, omniscient, omnipresent AI that decides everything For me, it. 


Keeichi Matsuda: Um, but you know what happens before then and even then, like, is there is 
that even a desirable state that I would want to just give up all of my data and everything, all 
of my all of the knowledge that these systems are getting? Am I going to want to give that to 
one particular entity? So this, instead of the kind of monotheistic model of Siri or Alexa, 
proposes more of an animist model for interaction with artificial intelligence, and we built 
that into a prototype called wall. Wall is, you know, running on an LM. And just basically you 
can we'll we'll come into your space. You can visit wall on your headset or on your phone and 
ask questions about the forest. And wall has a very strong domain expertise, as well as kind 
of a sense of character to hopefully make that learning experience, rather than being this 
passive thing that you just sit down and listen, something that kind of forces you to engage 
your curiosity about as well. So I hope these projects are interesting as a provocation. Um, 


yeah, that's what I've got. 
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Frode Hegland: Interesting. Thank you. Just before questions. Yeah, they'll come up. I just 
want to point out that what we saw now is real. But of course it's completely fake. Right. So 


how fascinating is that? It's like another meta layer of fantastic place. 


Dave Millard: So certainly the first thing is I wanted to thank you because I've been using the 
hyper reality as an example of where this technology could go for for at least five years now. 
So it's great to meet you. It's a wonderful piece of work. Thank you. There's a if you watch it, 
there's a bit at the end where the person gets their identity stolen and they have to do a swipe 
in the shape of a. Cross to rejoin Catholicism, which I think is absolutely marvellous. Well 
done. So but what I was going to ask was I've actually been using it as a warning, so I use it. 
I've been interested in the ethics of these these hybrid spaces, and thinking about going back 
to the idea of the edit button and those kind of elements. And this came up in that earlier 
discussion about, well, who has permission to do these things. And I think that's a wonderful 
example of what happens when we corporatize our space and where, you know, we have 
people coming into our personal spaces and our interaction spaces in a way. And that's kind 
of what that video communicates. So, so first of all, I wanted to to ask to what extent is that 
your intention with your work? And also what sorts of reactions have you had in those kind 


of veins when you've shown your work to people? 


Keeichi Matsuda: Um, yeah. I tried to kind of steer away from making it something which is 
either a utopian. This is the vision you should all do it, or the dystopian vision, which is sort 
of the same in a way. I find it was more interesting and got more interesting responses from, 
and provoked my curiosity more. When you try to to show something that just feels not like 
what you think it should be, but what it is likely to be. Or, you know, if we kind of follow the 
existing trends or currents, what what that leads to what's in people's mind. And for me, as a 
designer and film maker, it's very natural and easy for me to just make that jump without 
having to think, what have we currently got and how can we change one thing to be able to 
just put yourself in that world and to build a whole world around it, and then to be able to 
step back from that and ask, is that a world that is that what you're trying to do? Is that is this 
what you're trying to achieve with all of that billions of dollars put into it? And if not, then 
what is it? And if it is, then is that what we want? And I think for me, then design and 
prototyping becomes a very valuable tool to be able to ask that question and understand a 


little bit about that space before you make that billion dollar bet. 


Keeichi Matsuda: Yeah. So I think, you know, that's I guess the first in terms of my intention 


is really around being a provocation and not so interested in terms of actually my perspective. 
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I don't really want to force that on anybody. But on the other side, in terms of the impact it's 
had, I mean, that that film has, you know, it has got me into lots of interesting rooms, like this 
one allowed me to have lots of very interesting conversations. And hopefully as well, as you 
said, you're sharing with your students. Hopefully it's allowed other people into those ideas as 
well. So I still think that filmmaking and this kind of speculative design is still a very 
powerful tool for being able to just get people to, to, to engage with it on a very tangible and 
almost like a visceral level, like people, people have a reaction which is not just an 
intellectual one, right? They're like, oh, or so and sometimes they're like, I know how to build 
that. We're going to make $1 billion. Yeah. 


Dave Millard: I'd encourage people to, to watch them. They they are incredibly provocative 


and really very, very. 


Dave Millard: So yeah. So thank you very much. Thank you. 


Frode Hegland: We're going to go Brandel and then see something else. Just. Somebody 
mentioned space. I think it was you just now, Dave. You know, the question then becomes, 
what is a space? Because if you're walking down the same street as me, two different headset 


environments, it'll be two different spaces in a way. But so so what's a space we've done? 


Keeichi Matsuda: I've been thinking about this problem a lot. I start to think about it as 
subjective space. The idea that two people with a different set of filters or different devices 
could have two different experiences of the same environment, and there are different ways 
to slice that, right. We could imagine that everybody lives in their own bubble of reality, and 
everyone has a completely different set of things that they see. We can also go the other way 
and imagine that everyone sees the same thing, and there's no ability to hide the thing. So 
basically we just treat it in the same way as a physical reality. Both of those have their pros 
and cons. But I think again, when we were talking in as Fabian. Sorry. Yeah. And your talk 
around, you know, the I asked about the politics of it, you know, how do you decide who has 
the permission to do that? As soon as we enter into any situation which is shared, or if it 
allows any person to control what you see in that space, it instantly becomes political. Who is 
allowed to edit? Who is allowed to enter that space? Who is allowed to see that space? And 
not only does that bring all of the political problems that we've had with the use of space until 
now, and, you know, maybe people argue, we just we just use those same conventions. And 
it's like, I own this land, therefore I should own this space. You know, either way, you're you 


are, you know, excluding excluding people. So having a way, you know, I think that the 
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design of those kind of shared if we're talking about kind of shared computing systems, 
which I sort of feel like most of us are here, then we, I think, need to engage with those 
political issues very head on and try and decide actually not. You know, it might be like this, 
but how do we want it to be? And we can use design and visualization as a way of testing. It 


would be. 


Frode Hegland: To go with the Brandel, I think. Okay. 


Keeichi Matsuda: Brandel, I heard you. I heard you invited me. So thank you. 


Brandel Zachernuk: You're quite welcome. It's a pleasure to have you here. I'm really 
intrigued by what you were saying about your sort of withdrawal from the sort of 
instrumental use of technology and toward the agentive. Is that more motivated by a 
recognition that the instrumentality of of of buttons and switches and tabs is difficult and and 
to try to absolve oneself of responsibility of those things, or is it to do with a particular 
confidence or desire for the more agentive mediated things, and what you've actually seen in 
practice, in terms of the the capabilities that you get from it, is it is it is it leaving one or is it, 


is it moving to one or the other? 


Keeichi Matsuda: Yeah, I think it is difficult. I think that the instrument model is very 
compelling and has lots of very practical use cases. But I think Frode designer, we try to find 
kind of conceptual models that are as low level as possible, trying to find the most kind of 
natural interactions. And whenever we've seen great advances in computing, they've often 
been tied to the way that we engage with those technologies, the interfaces that we use. So 
when we move from punch cards to, to to command line to GUIs to mobile, each time, we've 
actually opened up greatly, like what we expect from that technology and what we think it 
can do. So in this case where we're entering this, this new realm and there's this other step 
change in it, it fills this world that it deserves a new, greatly kind of simpler and more human 
way of interacting. And, you know, this, this idea of, I guess you could you could talk about it 
as, as kind of, you know, utilizing natural language processing and it being a conversational 
UL. But I also like to, to, to think of it as communing with gods as well. The idea that, you 
know, if I want to go somewhere, I can pray to the God of travel and the God of travel will 
come to me and take me where I want to go, based on what it knows about me as well. So it's 
really. Yeah, I think the real, you know, it's really just the age old things of, you know, it's 
simple, too simple to use, simple to understand that we already have strong mental model for 


what it should do, time to task, you know, all that kind of stuff. But maybe it's not in 
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isolation. But I guess hyperreality was such a thing that people kind of used as a model of 
what the future could look like. I wanted to show something that was just a completely 
different route to that. 


Leon Van Kammen: Can I ask the next question? 


Frode Hegland: So. So hang on. What did I miss? 


Keeichi Matsuda: No, no, I just I'm just getting my close up, that's all. 


Frode Hegland: Oh okay. Okay. Right. You had. 


Leon Van Kammen: Oh yeah, a small question. 


Frode Hegland: Okay. And then we're over to the next presentation. Yeah okay. 


Leon Van Kammen: I have two questions. First question. Can you make a picture with me 
together? Yeah. Okay, fine. 


Frode Hegland: Just look normal. Yes. 


Keeichi Matsuda: So I'm interested. 


Leon Van Kammen: The the question is basically, um, in your last movie. There were these 
small beings or helpers or assistants or agency agents? Yeah, yeah, I have seen them. And I 
first of all, I'm a huge fan. I did not know that you actually made those pictures. Like also 
these like starting from the supermarket movie I think I've watched so many times. Oh great. 
And it inspired, I will tell you later all about it inspired a lot of stuff. But anyways. This last 
video of you was quite emotional. Like when it when it was sort of like ending. I had a mix 
of emotions there, and one of them was very personal because I. Well, this is a spoiler alert 
perhaps, but there is this guy who has some kind of it almost seems like he has some kind of 
super pate version, which is is one agent which can do everything right. And it's almost a bit 
intimidating to to the girl who has all this, this whole sort of almost a zoo of agents and. She 


was also having sort of mixed feelings like, well, what? Like he was like sort of like, oh, it's 
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so easy. Look, why why are you why are you. So you're you're making itself way you're 
making it way too difficult for yourself. Could you maybe tell tell a bit more about this 


emotional roller coaster? Yeah. 


Keeichi Matsuda: Well, I mean, you know, when you're making a film, it's, you know, you 
want to have a narrative that allows you to be able to address the core issue that you're 
interested in through the interaction of characters. And for me, in this film, it was the core 
issue was one versus many. Do you have one agent that is able to kind of crunch all of your 
data, take everything from you and serve you up a lifestyle or a way of doing things that is in 
line with, you know, the values of Corporation X, not that one. But you know, in general. Or 
do you want to promote a model where people can become producers as well as consumers? 
And so I think the character that you mentioned, the guy who has the one agent, is kind of 
representative of the the blind faithful, you know, evangelist of the, of the one God, whereas 
the, the other character, Maya, the girl, she's she lives in this pantheist world and she creates 
agents as well. And it's more of a community. And so I think it's not it's not so subtle. But 
that's thank you, thank you, thank you. 


Transcript: Sam Brooker 


Sam Brooker: Hello. And I'm going to be talking about modern railways and rituals at two 
different points. And then there's a bunch of other stuff in the middle as well. So I'm going to 
start off with some fundamental definitions. I think it'll be useful for what we're talking 
about. So we're going to take medium as meaning a means of communication. So in that 
definition all of these would constitute media or would sit within that definition. Whether 
you've got Paul Revere on the midnight run, riding across to passing information about the 
English arriving, whether you've got North Korean TV or you've got these like brazier things, 
which if you've seen Lord of the rings, Gondor calls for aid where he's very exciting. That's 
all along the South coast in in the UK you go and see them where they talking about Pravda 
or all of these various different devices over here. All of these are ways of collapsing distance 
to bring people closer together and to transmit information on a range which isn't available 
under normal circumstances. That's why we've got all of this chutney over here, because 
there's a lot of people who can't be here, but now they can because we're bringing them 
closer. So this sits under what James Kerry talks about as the transmission model for 
communication, which is a process whereby messages are transmitted and distributed in 


space for the control of distance and people. 
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Sam Brooker: I bring you closer to me. I give you information, information meaning to give 
form. And I shape your thinking. I shape your behavior and attitudes and your culture and so 
on. You can see the distribution of Christianity across the world as an example of this, with 
missionaries and with the kind of spread. And you can think about it from a colonial 
perspective as well. The main thing here is that that's one way and possibly an easier way of 
thinking about communication. There are other ways of thinking about it, which Kerry talks 
about as well, which I'll be coming on to. But before I go to that, I just want to talk briefly 
about technology. So if we're going to define technology, hopefully this isn't too controversial 
a definition and no one's going to jump. No way. Surely. Hopefully everyone's happy that I've 
accommodated everything that they do within these few short words. So if we're thinking 
about technology as the application of scientific knowledge and principles to create tools, 
systems, solutions and so on, I'm not going to give you examples of technology, but I do want 
to talk a little bit about technologies. And so when we're talking about technology, we have to 
decide if we're talking about the incredible new idea that somebody comes up with, or when 
that technology starts to impact on social processes, on behaviors, on culture, and start to 


kind of lead out into content and start to actually have an impact. 


Sam Brooker: If you do two timelines, one for when different technologies were invented and 
one for when they start to matter, they're usually about 30, 40, sometimes 50 years apart. So 
television sits around for ages before it's of any real significance, film a little bit faster. So we 
can think about technology in that term. We can also think about it in terms of by association, 
which I talked to a couple of people about already, where two different planes of thought 
intersect and create a new idea. So a particular technology, a creative activity, and that creates 
that lock in the middle of it. So if we're talking about the importance of technology or 
defining the significance of technology, we can think about it as postman talks about it in 
technology. I'm pretty sure that's from Technopoly not amusing ourselves to death. Pretty sure 
it's technopoly about the idea that the use is made of any technology are largely determined 
by the structure of the technology itself, so the functions follow from the forms. It's a kind of 
throwaway comment that's incredibly important and just sort of disappears into the flow of 
the wider argument. Essentially, the idea is that smoke signals can convey a certain kind and 
depth and complexity of information in the same way as the written word can or a computer 


can. 


Sam Brooker: You can go to Bart. And the idea that the chemists invented photography 
because they devised the process by which photography could happen, and then 
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photographers worked out how to use those processes. So we can think about this in terms of 
coming up with these functions. And then they define the forms and the shapes and the 
behaviors. That's quite a kind of techno determinist way of thinking about it, rather than a 
social determinants way of thinking about it. And you can also think about it, as Clay 
Christiansen talked about it, as the idea of disruptive innovations being technologically 
straightforward, consisting of off the shelf components put together in a product architecture 
that was often simpler than prior approaches. It's not about an incredible new technology 
you've come up with. It's normally a bunch of pre-existing bits that you jam together and you 
call it Netflix, and suddenly you're a huge billion dollar business and everything else is 
collapsing around you. So it's very rare that it's a new technology which completely changes 
behaviors and activities. It's much more likely that lots of pre-existing bits and pieces are 
pulled together into a useful and interesting thing that makes people get excited and makes 
people want to use them, Edison said. That we make we'll make electricity so cheap that only 
the rich will buy candles, and it's about the cheapness of the components and the ease of 


access, as opposed to the invention of the technologies themselves. 


Sam Brooker: So. In this respect. In some ways you think about things like VR. I'm trying to 
talk about VR for once, because I always come here and talk about hypertext, and I feel like I 
should stop in your direction. In that respect, you can think about VR as being a game 
changer. It's cheaper on the Apple side. It's better than existing hardware. Those are exciting 
innovations that go with it. I do have questions about it. One of them is what it's disrupting. 
We were talking at lunch about. With most technologies, you can draw a line between where 
it starts in theatre. Then it becomes, well, let's capture theatre and make it something you can 
watch. And then let's make the screen smaller and in your pocket. And you can normally 
draw a line between the home office of the 19th century and the Home Office today, and 
understand why these things go through these skeuomorphic processes and how they're 
connected to each other. If I try and work backwards from VR, I'm working into now like I'm 
in a world, and then you put a world on top of my world and I can't really see that causal 
chain. And that's one of the kind of questions that I've got. One of the things I'd like to talk 
about. My question is, is it a kind of magnificent platform in search of a purpose that we 


know it should be for something? Come on, everyone use it, please. 


Sam Brooker: But they don't seem to want to. But it's so good. Just get on with it. So this is 
where it brings me to Kerry's other form of communication, which is ritual communication. 
So in Kerry's words, if the archetypal case of communication, the transmission view is the 


extension of messages across geography for the purpose of control. So closing distance, 


444 


bringing people closer so you can put information into their brains and shape their thinking 
and behaviours, then ritual is about a ceremony that draws people together in fellowship and 
commonality. You learn nothing by seeing this cartoon about how bankers bonuses and 
etcetera and French bankers bonuses. But you do get a sense of, oh, we're all part of a 
common xenophobic community that hate finance. Or isn't it warm, warm fuzzies of 
xenophobia? So at the same time, when you have a royal wedding, for example, that's a ritual 
event. You learn nothing by somebody having a baby or getting married. But there's a feeling 
that you're part of a community, you're part of something. You're kind of pulled together. So 
the transmission model is about passing information to others, and shaping ritual is about 
coming together. I think about watching the Generation Game with my parents every now 
and again for Remember the Generation game, and like I didn't gain much from it except that 
it was a sense of family, a sense of community, a sense of coming together, a group of people 


who didn't spend that much time together. 


Sam Brooker: But we forced ourselves to come and sit and watch a not great game show on 
TV. So we're thinking about ritual. Ritual is this idea of coming together of people for a 
sacred moment, which is what makes this seem so ludicrous to me. When I sat and watched 
through the advert saying, come on guys, sit on your own in the dark. Yeah, and it's this one 
right in the middle where there's a dad playing with his kids, and he thinks, if only I had a big 
screen over my eyes so I could play with them more, better. Again, I think there's a lot of this 
kind of closing the gap. The most recent video of of meta showed Zuck and a tame journalist 
going, wow, it's like your face is in front of me, but your face is already in front of me. And 
so I get promoting VR on transmission. I totally understand that idea of we can have 
meetings all the time now. Don't worry, don't go to bed. Come and have a meeting. So I get 
the idea of closing the gap. That's great. And I get the idea of VR doing that because it does 
do that. 


Sam Brooker: But it's trickier to sell the ritual side of VR, which is not to say that it's actually 
impossible. I think that actually VR is great for these kind of ritual activities, and I think 
that's what they did really well with iPads when they were showing, you know, grandparents 
seeing children that had just been born on the other side of the world. Like, I definitely see 
the ritual argument for it has the potential to be this a device for the transmission of 
information, but also a mechanism and a medium for sharing and also for coming together. 
So I've kind of. Close this with a huge quote. No particularly good reason. Just so you have 
something to look at while I talk for a bit. Originally this was highlighted, but it disappeared 


when it switched over to keynote. So the two main parts here are the first line of each of these 
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first two paragraphs. That in one sense, when we're thinking about media communications 
and media businesses and so on, they're unique because they are our means of accessing our 
culture, our means of accessing communication, their how rituals take place, how you bring 
people together around a kind of common idea. So in one sense, media businesses are unique. 
They're complete. They're not something which you find anywhere else. And in other sense, 
they're just businesses. They have shareholders And almost so much done, no question. And 
so in another sense, they are just businesses. And I think often there's this conflict between 
the incredible new technologies on one side, but also their role as kind of communication and 
content tools and so on. So I think enthusiasts like those who build model railways or play 
Minecraft, just love the idea of something and want you to like it to come and see my model 


railway. Come and see my Minecraft world. And if you don't. 


Sam Brooker: You don't like model railways, or you don't like Minecraft or whatever they're 
enthusiastic about, it's quite hard to sell it to them because it's just cool. You want to say VR 
is just so cool, why don't you understand? And it's because they don't approach it from this is 
so cool. They approach it from what's it cool for? For me. So how do we get people from the 
wow of things like VR to the why of things like VR? And that's the question which has words 
that kind of rhyme a little bit, which is a good way of finishing that I'm going to leave with. 


I'm done. Exactly five minutes. Well done. Me. 


Frode Hegland: So before questions I will address that in one specific way because it's a very 
provocative and it's absolutely perfect question. Um, the whole what is VR we spent a lot of 
time on and now we don't care. Now it is the things in VR. So I'm interested in how to have a 
graph on the wall over there. Vr in itself is irrelevant. And this was very much taught by 
Fabian Nosek. Here's a projector that's also. We're talking about extended environments. So 


the shift from the medium to the things in the medium is becoming interesting. 


Sam Brooker: So you really should have moved. You had a chance. You just. 


Gabo Beaumont : Um, I kept I kept back and thinking, okay, you're saying that the the tools 
shape the way that the tools are happening. Also how we communicate and what are the, the 
the ideas that we share with others. Right. But early on, we were like taking this obsidian tool 
and, and Frode said that it's it's interesting to also think that maybe the tools that are shaping 
ourselves to how we communicate with each other and how that helps us to communicate 


with each other. So I don't know, how are your thoughts on that part? 
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Sam Brooker: Yeah. So that's that's one view as well. We were talking a little bit about the 
kind of mall approach that's often taken that, you know, you put people into a mall with a 
bunch of apps and you say, this is a virtual world. And what kind of forget is that? The initial 
stages were that you kind of slowly lowered people into it. You got them excited about being 
able to take photographs and be able to see them straight away on a screen, and then you very 
slowly integrate them into an ecosystem and you slowly bring in the other kind of 
applications and so on. And so I think there's a kind of slow process of integrating people into 
things where people do start to accept and engage with and absorb things. But we were 
talking a bit about the Everett Rogers Graph. I think that's such an interesting point that we're 
at right now, because I think most people know what this technology is like. It's not a 
mystery. We're not in the early adopters necessarily. We're in a point where either it becomes 
universal or it becomes a kind of interesting hobbyist activity. And so I think this has now 
handed over to us. It has handed over to people, and we're trying to sell, get getting involved 


with it and getting excited about it. 


Sam Brooker: And I think it is still something which struggles to go outside, people who just 
get it immediately. And it's how you get that bit where you say, here's why it's better than 
because what I get from a lot of people is they say, well, why not just use a keyboard? 
Keyboards are great. Why not just use a screen and screens are great. I've got a printer. Like 
we were talking a bit about office 365 and how the nobody tell anyone but London Fire 
Brigade is entirely made up of people who download files, edit those files and email them to 
people because they just don't like the idea that a file is just nowhere. They like to have it in 
their hands, and ideally it would be a printout, which is what everyone does. They print off 
the PDF, annotate it, and then hand it to somebody. So I think there's a real long tail of this. 
And if you're trying to just make a big push, it's really hard and really hard to convince 
people to get over that initial, well, what's it replacing? Because what you're saying is it's not 


really replacing anything. But if you all use it, we'll work out what it's replacing. 


Gabo Beaumont : One last little question. Is there any technology that had this big impact like 
you were just describing like, okay, here's the technology. Like I just imagine cars. Hey, this 


is the car. We can use it. Telephone all the for all the forces. Okay. It wasn't weird, I think. 


Sam Brooker: I mean, it's interesting ordinarily with these kind of things, you see a long 
crossover process. So we're like the magic lantern, for example, which was a kind of 


Victorian parlor entertainment where you would project light through a glass slide and it 
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would shine onto a screen. The crossover between film and Magic Lantern is actually 
enormous. It's about 50 years where magic lanterns are still being used in various different 
ways, particularly because they were considered to be superior because film couldn't do 
color, for example. So there's a whole crossover period as one technology replaces another 
technology. And I think that's one of the challenges, is it's hard to integrate this into other 
aspects of daily life. If I'm sitting watching TV with three people and one of them's got a 
headset that's got their subtitles right there, they're like, you guys don't even know. My 
subtitles are so good. And I'm like, yeah, but I'm watching TV with my friends and eating 
Pringles. Like, unless your headset can put the Pringles in my mouth, I don't see what I'm 


getting from it. So I think there is a. 


Frode Hegland: Opportunity. 


Sam Brooker: One day on pitching ideas. So if anyone wants to launch my Pringles device. 


Sam Brooker: I would just say I think that I mean, I think that's one of those tricky things. I 
mean, the crossover between like, horses and cars is also huge. And even with the model T 
Ford, which is a good example of disruptive innovation, there's still a long crossover where 
you still have a whole range of different other conveyances which which slowly replace the 
vehicle. So I don't think it's ever that simple. And I think it's just accepting this, this. Yeah, 
exactly. Accepting this process and accepting that for a long time it's a novelty. And then it 


might get integrated when people work out how it gets integrated. 


Dave Millard: Yeah. Thank you. I had a double espresso earlier on, so now I feel I could talk 
to you at your own level. So. Yeah. Slow it down for us. Um, yeah. So. So I thought it was 
really, really, really interesting, very provocative. And the idea of kind of those ritual 
interactions kind of captures, I think, some of the unease around VR really nicely. I was 
thinking, though, about the kind of the wider space in particular. So I did work in mixed 
reality and in mixed reality of the potential there that you have a technology that enables new 
types of shared connection, because what you can do is you let's say you're playing a mixed 
reality game in a space with a group. You and the other members of that group are in a, in a, 
in a quite tangible way, in a different space than other people there. So the technology is 
facilitating your connection in a way. Yeah. And so does that follow? Is it do you think there's 


potential in the broader sort of spectrum of doing those kinds of. 
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Sam Brooker: I think that's a really that's a really good use case for it. And I think it sits quite 
nicely in that kind of slightly old fashioned vision of games as kind of defining a space within 
which there's a set of rules. And I think that's always been a great way of thinking about it. 
And I think you could probably scale that to other spaces as well, because it's been talked 
about as a model for, you know, judicial process as well, that within a judicial context, there 
are some things you can take into account and some things you can't. And I think if you're 
going to define a space in which that interaction takes place, I think that's great. I think the 
challenge is to model something which people want. And I think one of the difficulties, 
particularly in this kind of space, is when you ask people what they want. We were again 
talking about this at lunch. What is it that you want to model? Somebody was saying, you 
know, would it be useful to be able to look at somebody and see all their interests and their 
favourite films and that kind of thing? And I feel like that's like for people who find dating 
uncomfortable, wouldn't it be great if I could just like, cheat and like, look at all your social 
media, find out your favourite film, go, hey, do you like my examples? Greek hip hop? And 
you're like, yeah, I do like Greek, you know? So it's whether what you're modelling is a way 
of kind of bypassing difficult social interactions, or you're enhancing an experience that 
everybody wants to be part of in the first place. And again, that's this idea of a kind of 
universal technology versus something which has specific applications. And I think that's 
kind of what you're talking about, which is moving into where it sits and finding people who 
can get excited about one particular thing it's doing, rather than all the things it's replacing. 


Make sense? 


Frode Hegland: Yeah. 


Dave Millard: Well, you have to sit on this question. 


Bronnac Ferran: Yeah. It's it's the. 


Sam Brooker: It's all firstly because it feels like it's getting such weight as you walk over 


towards the special chair. 


Bronnac Ferran: Um, I really like your idea of the ritual. I think you're onto something there. 
And what entered my mind was this summer in my local park, they had this Pokemon 
convention. So people came from all over the world, and they all got into a local park, which, 


you know, got some protests from the and but they are literally I'm walking through the park, 
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walking around just like this. So basically they're participating in a ritual. Yeah, it goes to the 
mixed reality, but that is it's ritualistic enough to stimulate that. If next year someone says, 


but actually we're going to use headsets, I think your catchment audience. 


Sam Brooker: That's a great example. And I was thinking about, you know, that thing about 
golf being a good walk ruined. But I think it's one of those things that people need a reason to 
go for a nice walk with their friends. So you go golfing. So I think there's a way of thinking 
about it that as a kind of enhancer to experience. And I think that's I mean, that's a great 


example of exactly that thing. Yeah. 


Bronnac Ferran: And it was struck by you mentioning the royal and talking about it as we if 
you're not actually a royalist, it's not something that. Yeah, it's absolutely a delusion. Yeah. 


But again, it's that thing if you've got enough people to subscribe to something. 


Sam Brooker: And I think the interesting thing with Kerry is that he talks about the idea of, of 
being anti-royalist as being ritual in itself. Like I think there's definitely, you know, as 
somebody who spends a lot of time in left wing circles, having left wing conversations, like 
there's a slightly performative aspect of being somebody who says, well, we all hate this, and 
it's quite nice to be part of a group because you feel so hemmed in sometimes to be able to 
say, well, I'm sitting in a room full of people who all agree with me, and that feels good. So I 
think that that ritual doesn't have to be around something that is, it can be around something 
that isn't as well. But but no. Really. Yeah. That's great. Thank you. 


Frode Hegland: I went to the coronation, you know, really separate conversation. Hang on 
Mark, you're coming up to Brandel. So we don't forget the online, but we have to really, you 


know, spin the wheels now. Brandel, please. 


Brandel Zachernuk: Um, so from the perspective, you can hear me. Okay. In the room now. 
Yes. Terrific. That's amazing. So from the perspective of, you know, there being some 
technologies that appear to be clear references and replacements for other things versus 
things that appear to be sort of solutions chasing problems. I can say, hand on heart that that 
analysis, even from the interior of the organizations that are responsible for creating those 
things, can only be happen after the fact, significantly after the fact. So iPad, iPhone, Apple 
Watch, all of these mobile phones and things like that. Um, if you look at the original 


messaging, the original positioning, their accurate reflections of what people thought they 
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were going to be at the time. And so it's actually never the case that there is some kind of 
preordained path that these things are on, and the sort of priorities and values that they come 
to, to occupy, so that those aren't necessarily forthcoming within the virtual reality space is 
not a unique indictment of it, but rather a reflection of the point that we are in history with it. 
Um, but I am excited for that solution for for that sort of process to, to kind of become further 


resolved. 


Frode Hegland: And you're the only one in the room who actually has the headset. So we 
have to bow to your expertise. Mark, you had a quick question. And then, Fabian, you get 


three words or less. 


Gabo Beaumont : So so a quick observation. 


Mark Anderson: More. Think back to this thing about saying, well, is VR a solution, an idea 
looking for something to do, an alternate publication? Because of time spent in the Cabinet 
Office, of a government with a group that were doing retention of documents, and the fact 
that the first digitally native documents have reached that point, and there is no process that 
doesn't depend on a print era to do it. And now we're drowning in vast quantity of data. When 
you look at things like space environments shown earlier, maybe, just maybe, that's one way 
that we can actually. So maybe it's just arrived at the right time. We don't know exactly what 
it is, but it possibly is a solution to something we really need to do and we should have 


started doing earlier. 


Sam Brooker: Thanks very much. 


Fabien Benetou: I'll obviously cheat, but it's okay. No, I gave a workshop last weekend with 
kids. They were 11 to 12 years old. We did VR. The whole ritual was let's play. That was it. 
There was no, like, further thought it was let's play. Let's do the thing. I'm not saying it's 
going to might be like general thing, but the mindset of let's just go at it, let's try it. It's 


completely different than in other contexts. 


Speaker8: It's more than three words. 


Fabien Benetou: That's like, I don't know how to count. 
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Frode Hegland: Thank you. 


Speaker8: Counting in English or French. 


Frode Hegland: Now we're going to move over to one of the very latest members, if I can 
loosely call it that of the community. He will be presenting on his own screen. As you see, we 
are going to post 30s for coffee. I completely agree with that priority. Coffee and water. 


Anyone else want to. 


Speaker8: A whole minutes mixing in person? I'll be. Okay. Over here. I was seriously. 


Speaker10: One. One question. 


Speaker8: I'd like. To find the. 


Speaker11: I don't think they can hear us, Teresa. 


Speaker10: Okay. 


Speaker8: So we can hear. 


Speaker 10: It. 


Gabo Beaumont : Yeah, we can hear. 


Speaker10: We were wondering if we can present soon because we have to leave earlier. 


Frode Hegland: Okay, so we asked earlier if there was anybody online wanted to present, and 
we didn't see anything. 


Speaker10: Yeah, because we read it later. We just answered. 


Frode Hegland: That's fine. You can go straight after this presentation. Will that be okay? 


452 


Speaker10: Yes. 


Frode Hegland: Now that we have a speaker and everything. Okay. Fantastic. 


Transcript: Frode Hegland 


Frode Hegland: So I'm going to talk about two things: Visual-Meta and Author and Reader. 
So, metadata we've talked about many times today and I just want to clarify how I define 
metadata. It is something that says what data is. You know, all documents have a title. It 
doesn't mean the system knows it's the title. So the metadata just says this is the title. 
Metadata can get really messy. All kinds of ways of looking at it can go all over the place. So 
in this short presentation, I only want to talk about document metadata, not tapping phones 


and things like the government might do. 

Brandel Zachernuk: The remotes don't have your screen. 

Frode Hegland: Uh, what am I not sharing? But is that what you said? 
Speaker3: Yes, that was Brandel saying. Couldn't see the screen. 


Frode Hegland: Thank you. Brandel. I love you. Okay. So I'm just going to go back to that 
slide then you can see now right. Brandel? Yes. Did he fall asleep? Yes. Yes, yes. Okay, okay. 
So just running through this author and visual meta metadata is what the data is. It can get 
messy. That is why I'm just going to talk a little bit about document metadata. Now we're up 
to speed. So that is what the document is. And that can mean how the document can be used 
and how it's described. So. What the document is, is inherent metadata. It is a heading. In the 
document is the heading of the document. It has a heading. Here it is. This other stuff is 
appended metadata, how it can be used, legals and all those kinds of things. And also 
described somebody saying, this is important. I don't care about those second ones because 
they are expensive. Like, you know, they're fair enough, but they're just out of scope for this. 
So what the document is, is how it can be addressed. We talked a lot about that too. That 
means with a normal book, you go to the library, you say, here's the title or here's the author. 


That's one way of addressing it and URL is another, and so on. And how it is structured, 
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where the headings are and what data it contains. You can have a document with a graph in it 
that has no computational meaning because it's a Jpeg, or you can have something that can be 
extracted and used by anybody who can access the document and then importantly, how it's 
connected, who it references and sites and links to externally and internally. Here they are. 
Oh, did you put your drinks in cups too high? What's drink them. 


Frode Hegland: So all your missed guys is the basic definition of metadata in this context, 
which is. Data that says what data is, particularly what documents. So particularly for 
document, how it is connected is a huge topic. But it's not just links going out. It can also be a 
glossary. So it's connected internally. You know, we could have a seminar or symposium just 
that. So the approach that I've taken and there you are Dave. It is just too it's a bit 
embarrassing because, you know, you're my professor, but my entire PhD, which has been 
running over time, it's supposed to finish today. So that's an issue actually, today it's just to 
take the document and write that the back, what it is. That's my entire PhD, and it's 
embarrassing to say it that quickly. So I have to put an acronym in, I put BibTeX, that's the 
format that we use. It could be anything. It could be literally crayons, you know, as long as it 
can be read by computer and person doesn't matter. So this is what you have in the beginning 
of many books. This is normal. Print this page. It is the metadata. It says what the book is. So 
visual meta is just this. So that's the whole point of what I'm going to go through. There is a 
ton more here, but essentially it is just that. So this is a visual metaphor from an actual 


document. 


Frode Hegland: So you can see the type. The font is tiny because there's no point having it 
big. You're not going to read it very often. But if we zoom in you can see it is exactly that. 
Plus more stuff. Any questions on this? No. Brilliant. Fantastic. So the really important thing 
here is this is already in the manuscript, right? You save with the title and you have a name. 
You usually don't. Unless you change computers with someone, your name will remain your 
name. So all of these things are there. This costs nothing. It's called export and it's done. So 
now we get to the demo part. There's two pieces of software author and reader, both for 
macOS. They cost money. Send me an email. I'll send you a code. If you're a mac user, you 
get it for free, obviously. So we're going to be going through copies of citation. What I think 
of as being powerful views, library function a little bit and then interactive map and glossary. 
The whole interactive map and glossary is very related to what Mark and Dave was talking 
about earlier. So copious of citation. It is as simple as this. We open a document in the library. 
Oh, this is fantastic seven hypertext and it's got visual meta. So let's go back and control. 


Click copy to cite the document. 
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Frode Hegland: Go to Author. Paste it and it pastes as a citation. So you click on that. You can 
see it's got all the citation metadata which was in BibTeX. We export that. Make sure it's the 
style we want. Open that up into reader. Go full screen, smallest document in the world. Click 
on the citation and it brings that in to show you right there, which is a phenomenally 
powerful thing. And Mark Anderson was your idea. Thank you. And it was a bit quick. But 
what I also did, because this cited document is also my library, I could click on the title and it 
opened that document to the page cited. So I've been doing this for a couple of years and I'm 
really, really proud of that functionality. But it is just for my software to my software. And 
I'm not exactly Microsoft, so it's a little bit limiting. And then over the last few weeks, I've 
come to the realization that we have big tech in. We can import big tech. That's how we do 
some of the resolving. How about big tech out? So that means that now if you paste it into a 
tech aware piece of software, it's formatted as BibTeX. So it means it's open for anybody 
working with that stuff. And that's just so exciting because visual media is based on BibTeX, 


it contains more than BibTeX. So that's why we can produce. 


Frode Hegland: So to me this is the biggest thing that's happened all summer. Okay. It is. You 
know, as I've said many, many times, my mentor was Doug Engelbart. And because of him, I 
got to know Ted Nelson quite well. And Ted is an absolute genius. He invented worlds. But 
the problem with inventing worlds is you tend to live in them. You know, he doesn't like the 
import export thing. So when I see a little thing like, holy moly, big tech connects with big 
tech, it's very exciting because suddenly we can work together, right? Powerful views. This is 
one of our future of textbooks. It's massive. So let's view it by color. So here we have the 
headings. And then we have color coded for the different articles. And it's based on whatever 
the user wants the colors to be super simple. It's a bit of an experiment. Also we can select 
text and we can do command F to see all the occurrences in the document that can be useful. 
And we can do things like see only the names. We always retain the headings because they 
are the context. And then now we're seeing all the glossary terms. Glossaries I think are 
really, really important because that is the author's way of saying these things are important. 


So here you can see where they are on the document. Hello, Olivia. 


Speaker3: That's you. 


Frode Hegland: So now I'm going to show you briefly folding and navigation. So folding a 
document is my fancy way of showing an outline. It's the same font for the heading is in the 


outline. That's really important. It's sculptural rather than off to the side. But also what I'm 
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doing now you can do right? This was too fast. Let's go back. So that's the folding. Just 
instantly go in and out of folding the document. But the other thing you can do right and left 
arrow keys goes page by page. But these books are like 800 pages long and some of the 
articles are very big. So if you do arrow key down it goes to the next level one heading. 
We've been experimenting a lot, and Fabian made me do a whole wall of front first pages of 
articles. You know, just to see what is the best way to find out where to go inside it. And we 
realized we already have that. All we needed was the navigation mechanism through which 
we can see those. So we built a basic library into reader. And yes, I'm selling features of my 
software, but the reason I want to show you that is this is based on visual media. So it isn't 
any fancy behind the scenes like reference managers going off and getting stuff. All of these 
documents know what they are, plus one thing and that is if you open a document. And 


highlight something when you search. 


Frode Hegland: It'll also, as you can see at the top there and there, it searches for things 
you've highlighted in the document. So in this case the word hypertext was highlighted. And 
Mark and Dave's article. That to me as a student is huge because it's often, you know, 
annotate stuff. Why? I can never find it again. Right here. I can find it again. So a different 
aspect. Now this is the map of defined concepts. So this is something I'm sorry Mark and 
Dave you're getting very heavily referenced. But when you explain that there's hypertext as a 
tool for thought and hypertext for knowledge representation, that allowed this to breathe a lot 
more. And I'll show you all why. So what we do here, we select Hamilton. Why not control 
click define concepts and we just write here's great all of that stuff. The musical is great. 
Then we click on map at the bottom of the screen. And here's Hamilton with all kinds of 
things. You can choose whatever name you want for this map. I just call it basic map and 
nothing is selected. No lines are visible until you select, and when you select and point to a 
line, you can see why it's there. Same dialog box, you can see why the connections are there, 
right? Okay, so here Doug has been tagged as a person and that is why he is an italic. It's 


supposed to look like a signature, so it's a little bit of a visual cue. 


Frode Hegland: So if your keyboard select all the stuff of one category to hide them and then 
bring them back again, they're selected. You can manipulate. This bit is just to remind you 
that if all the lines are shown all the time, it becomes a complete mess. So that is hypertext as 
a tool for thoughts. And now we're done. We're going to turn it into the generated PDF. And 
now it becomes a glossary. So from defined concepts to glossary, because now we're reading 
and a glossary is a completely different thing. So here we are. We're in a PDF. We select 


Hamilton and we do search the document. These are all the occurrences as before. On the top 
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you have the definition. And you can click on any of the bold words to jump to them in the 
document. So you can you have conceptually the same mapping. You just don't have my 
crazy layout because it's actually useless. That may. It works for me, but not for anyone else. 
I just want to show you that again from a different way. Select PDF command F to find. And 
document and you can see here it is on top. Click on these different things and have fun. But 
it can also do command shift f find in library. So it searches all the document for that text. 


Come on. That's pretty. 


Speaker3: Cool. 


Frode Hegland: It's new to me. It's the coolest thing we have. And of course, a little I and I 
can do amazing things for so many different eyes and legs. My main advisor, supposedly he 
says it shouldn't be called artificial intelligence, but advanced it. And he's right. It's just a 
package of so many different things. All we're done here select text. Even if it's badly done. 
Control. Click summarize in a sentence and you can write your own prompts and it gives it to 
you. How to actually use this is quite a challenge because it's not so quick and easy. And all 
of them start with something like this article says. You know, all just reading that is a waste of 
time. You know, it's an article and you know that to get the prompts around that is a 
challenge. So going towards the end issues that we're dealing with now. So this is stuff we 
just don't know. One of them is as simple as color. As we were talking about earlier, you 
know, humans don't often stare a lot, you know, so to looking at text for a long time can be a 
bit of a stress. That's normal view. But this some of these lines now have been selected and 
highlighted. You can see a faint gray point is to not interfere too much. So we have a 
command already that hides everything in the document except for what is bold and 


headings. 


Frode Hegland: This one also keeps what has been highlighted. So from my eyes this works. 
While working on a long document I can see these three paragraphs. These are the things that 
I want to highlight for myself. Key is when I export this document, this is not included so I 
don't have to worry about reformatting and all of that stuff, right. This is not been 
implemented yet. And also things we're looking at is the color. This is the normal view. This 
is the warm view. And by testing this it's literally just staring at the screen for a while and 
then asking different people to do the same. This is a warm dark mode. This is the not warm 
dark mode. And we're looking maybe to lighten that a bit and then maybe go really, really 
warm on this one. It looks kind of stupid, but when you actually have it in your screen and it's 


a big one, the orange tends to mellow down a little bit. It becomes useful. And this is the 
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other of the two issues. So graphs are great. And once you're going to have a huge graph in 
your room that I'm hoping we are, we start having an issue of nesting graphs somehow how 
that's going to be visual. And I'm an intensely visual person, meaning I don't want too much 


screen junk. 


Frode Hegland: So I'm wondering about simple things like maybe the word affordances. I put 
little hard brackets on the side, and that's maybe enough to say click on that and it opens 
something else. I don't know. I don't want to just have blue underlined links on here that 
already have their ideas. This is the kind of stuff that is detailed, but I think it's important for 
us to talk about it because whatever will be used will be used forever, right? In closing. Kind 
of. That was author and reader. You want to have a little bit look at ideas for radar and XR. 
This is obviously a complete fake mockup. So there's reader the little R right. So the first 
thing we found, I'm not sure if Adam Braun is still here, but one of our future techs lab. The 
people here who meet every Monday. Swedish guy. Very clever. He also does some work. 
And one of the things he started playing with just reading in a VR space, and then he locked 
the document to your view so it's not in the room. And he found that was really comfortable. 
You can lie back and you can read. So when people say you're not going to be reading in VR, 


I think that's absolute nonsense. You know, in 1 or 2 generations of headsets, we will. 


Frode Hegland: And this is really easy. Apple has already let me do this when the headset 
comes out. This will work. But what about if I want to have a graph behind it, you know. And 
how would I pull that in and out when I need it? Maybe sometimes it helps me think. Maybe 
sometimes it doesn't. Also, and this is why it's so important to talk about dreaming, because 
this looks very rudimental and simple. But in terms of what Apple and other developers give 
me, what are these? Are they windows or documents? Will the system allow me to have three 
documents like this that once like this? And if it does, what are the you know, how can I 
move them? You know, is it a grabbing? Is it a pinching? Super simple stuff. But if we 
together don't look at it, these big companies are going to design decide how it is. And here's 
one with the even more stuff around it. So I want to say thank you. I wasn't planning to go 
last, but since I did go last, I want to say thank you. Thank you again for being here. It's been 
amazing. Now we're going to have some alcohol and then some dinner for the ones who can't. 


Thank you very much, everybody. But no wine before questions. 


Leon Van Kammen: Thank you very much. This is Leon. I have a question about this visual 
meta. This solves a big problem, a very old problem, actually. What people have in the web 


browser or in even a text editor, for example, large documents were sort of like limited to 
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control or command F, and this is the only tool we sort of have. And this is for me, if I look at 
this visual media, it would be great to sort of copy paste it. Like if I have a text version of a 
PDF document or an HTML version, if I could just copy paste it into the comments section of 
an HTML document or in the bottom of the text file, like in theory, we could sort of do the 
same outside of reader. Is that something you were also envisioning? That this is just the sort 


of like example of of how this could be used? 


Frode Hegland: Yeah, absolutely. I'm just. Oh that's disconnected. Okay. I'm just going to. 
Yeah. I'm just going to turn this. Yeah. I mean, the whole thing, this is, you know, thank you 
for your very leading question. Obviously, this is just an approach and it comes from playing 
video games and reading an interview in edge magazine many years ago haven't been able to 
find it, which goes back to the issue of citing a developer. He was talking about the fact that 
it's a huge open world game and it's amazing you can do this and that and so on. And they 
talked about the AI even back then, and he said, it's not that difficult to make the other 
players smart. What we need to do is make the environment be aware of what it is and be able 
to communicate that. Use the example of a piece of wood. Piece of wood will have to know 
how heavy it is. If it's transparent, how much damage to break it, it can be set on fire, it can 
be picked up all these things. Right? So once every object in environment knows what it is 
and can communicate this, you can do what the hell you want. You can drive a tank through 
the building, you can fly it, you can do all these things. But what we're missing in our work 
environments is that things don't know what they are. So yes, you're completely right. 
Anything should be able to have this. 


Frode Hegland: And when it comes to XR, one of the notions we have is to have an appendix 
to the appendix. That is temporary. So if you take this document into whoever's room and you 
do all kinds of things and all the elements and picture her 3D whatever, and then you leave it 
writes another appendix describing the scene. And I think this is where what you're working 
on would be amazing to write that in the back, because all it needs to say here is Leon's way 
of looking at the world in 3D, right? It has to say who was referring to or what system, so that 
any other system that can read that it opens up and everything is in the right space. So it 
shouldn't matter what it is. And of course, this also goes with images where we're talking 
about hypermedia and how you can't just link text. You can also should be able to link 
images. We used to have image maps. Right and image maps were amazing until they were 
killed because of whatever. But why not have the entire image map hidden in the file name, 
right? You don't even need Exif. So that's one way of writing at the back of the envelope, so 


to speak. So just say what things are and we know we'll know how to deal with that. One 
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more question and then you get one. Come on. Anybody but Dave can ask one question. 


Okay, well, okay. Okay. Fabian. 


Fabien Benetou: It might be an annoying one in the sense that, well, first, it makes sense to 
me and in the perspective of having the edit button there on the table, I want the table to 
know where it is and be able to then build on top of that. So it makes perfect sense to me. 
And yet in let's say, a hacker mindset, things are what they are. An infinitely more okay, I can 
use the table for what it is, but I can use it. I can flip it around and use it to juggle what I'm 
doing. So? So it's. It's not. Yeah, I think there is part of it that's I don't know, it's delegated to 


the user and their own creativity. And it could be. Yeah I don't know how that works. 


Frode Hegland: I can tell you the answer to that is we don't need to know how it works 
because it won't be the same everywhere. You will want to be going into environments where 
you have no control. You know, like at the pest control, at the airport, passport and physical. 
You should have very little control, right? There will be virtual equivalents of that where you 
should have very little control. So there will be different. You know, the virtual world isn't a 
headset. And that's what it's so nice to hear from the different discussions. It will be entirely 
different things and entirely different settings. And here, you know, this is not a VR 
conference. Of course, it's the future of text. The idea that really struck me when I did the 
first future of textbook was text isn't a thing. Talk about the future of text is quite stupid. 
There will be future of text in poetry, a future of text in academia. You know, there are so 
many different types of texts. Same with VR, but within this community, I hope that we can 
work more on work in exile, which will be partly with text and partly with other things, but 
we have to define. The thing. All right. Thank you very much, everyone. And I will now sign 
off. 


Transcript: Bronnac Ferran 


Bronnac Ferran: I didn't have time to prepare the presentation that I wanted to, but Frode said 
that maybe I should just say a few words. It sort of would have fitted better, I think, with this 
morning's series of presentations. But I've just recently finished a PhD about post-war poems 
and texts in print called Auto Poetics, and I was looking at the emergence of non-linear and 
digital tendencies, actually in printed formats that you can see within concrete poetry and 
visual poetry. So I wanted to reference the first theory of text that I know about, and it was 


created by Max Benzer in 1957, and he kept iterating it, and he was talking about non-linear 
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texts and various other texts, and he borrowed that in a way from Francis Ponge, who was a 
French prose poet who first started talking about the text in 1949, and how the text is actually 
differentiated from what he called the literary object. So the literary object elevates things, 
whereas the idea of the text is that it could flow under and over and in between things. And 


this is from 1958, visual text and exhibition in Stuttgart. I'm going to. 


Frode Hegland: Change to this camera here and for everyone, so you can more easily show. 
That would. 


Bronnac Ferran: Be great. 


Frode Hegland: Uh. That's this one. Then and. 


Speaker3: Yeah. 


Bronnac Ferran: So the idea of the visual text then really took off. And Porsche was saying 
that in the 50s generally, you know, with television coming on and advertising, the visual 
perception started emerging as a field in its own right and visual design. So within the field of 
poetry, you start getting a linguistic turn as early as the 50s, particularly in German post-war 
language works. You get kind of decomposition of the poem itself from the word into the 
sign. So you get this kind of work emerging, which is really semiotic, and it's still being 
called visual poetry. Um, you start getting frames there over frames, all of which are based on 
letters. Um, I haven't really got time to go into this too far, but I wanted to just show you 
some other examples of what I'm talking about. Um. 


Speaker3: So. 


Bronnac Ferran: And you can trace some of this back, of course, to futurist experimental 
typography and Dadaism. You get a more programmatic approach to the visual poem in the 
60s, Futura typeface, which is based on a zero and one line and circle, starts to be used 


consistently. You get this kind of textured effect, all built up from one single letter or two. 


Frode Hegland: Here as well. 
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Bronnac Ferran: That's called cold type structures. So what you were getting was a kind of in 
a way, it was a typing. It was a printing revolution as well as an electronic revolution, 
because letterpress and hot type hot metal type were suddenly being superseded by electro 
photocomposition using light based printing. You start getting different sort of multimedia 
effects. And this is the kind of work that I was looking at in my thesis, is all built up from one 
character. So it's in a sort of pattern language emerging, which is, of course, very much what's 
coming through now in AI. And the whole idea of the attention mechanism and the vectors, 
you get all that coming through in printed formats. And I just wanted to sort of draw attention 
to this a little bit, because it's clearly foreshadowing what happens later, which is very much 
associated with digital as in digital, digital and electronic literature. But you're getting. This is 
from 1965, poems that start to get moved, and they start to gradually move off the page into 
environmental context. So you have like poem machines start getting made that turn around. 
Then you get works of public art using poetry projected into different public spaces. So I can 
write this up for the presentation, but I sort of wanted to flag it up because I think it's been 
kind of overlooked a little bit. 


Bronnac Ferran: And what I what you find in reading back into some of the theory of texts 
that Benzie was writing, along with Haroldo de Campos in Brazil and Haroldo de Campos 
wrote an essay called The Open Work of Art as early as 1955, and he's talking about this 
relational, cyclical text, tracking it back to Mallarmé and Dru Joyce and Max Benzer in 1959 
was writing literally about digital tendencies within the poem and within the work of 
literature, which they were talking about was an atomization and a fragmentation of the line 
into different elements. And once you got that fragmentation and decomposition, then the text 
becomes mutable. It can be moved in various different directions, and you get a non-linearity 
emerging off the page into the environment. And that also shows us that digitality starts. It 
doesn't need computation. Computation is just one way of taking that forward, but it is 
actually linked, I suppose, to human brain. So I could go on about all of that, but I'm going to 
stop now because it's too long to expand on, but happy to answer any questions. That was 


not. 


Frode Hegland: Very long, but that was fantastic. Second to last presentation today. And 


finally someone talks about texts. Not done yet. 


Frode Hegland: I'm sure there's another question or comment. Yeah. Please move over. I just 
want to say, you know, we are all about the stuff, but we make it very clear, as Fabian 


showed, both making stuff and paper. You're showing how even before these technologies, 
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paper was being expanded and it was using technologies of different print styles and so on. 
So I'm very grateful for what you showed and for whatever you want to submit as a paper to 
put in the book, even if it's just a few, even if they're horrible photographs, just anything. 
Please. 


Bronnac Ferran: Because the other point I'd make on that just in response is what I didn't 
mention is that once you look, I mean, the person I did my PhD about did use hot metal type. 
So if you look at something like this that he did from 1965, this is an overprinting of words. I 
mean, it looks like one of the spectrum synthesis that you were showing this morning. I 
actually wrote about this as a synthesizing process, but he's actually used top metal type, so 
you can still feel the indentation on the paper. So you get this haptic quality about this 
material poetry, which I think we're trying to get back to. I mean, a lot of references today is 
being about how do we get this multisensory dimension. And clearly there's a lot of research 
going on in haptic areas. And I think we have a lot to learn from simply engaging with this 


kind of semiotic, semiotic poetry. 


Leon Van Kammen: Hello, this is Leon. Thank you very much. I think this is a pretty mind 
blowing experience for me, especially when you started to talk about these poet machines or 
these dynamic poems. Can you maybe tell a bit more about that? Because what I think is very 
sort of it's a very actual topic almost because it's it feels almost the same as like you have like 
this generated text or this GPT stuff, which freaks some people out and some people love it. 
Like, did that maybe already happened back then when people came out with these poem 
machines like were, let's say, the traditional poems upset by this. Like, can you tell a bit more 


about this, this, these poem machines? 


Bronnac Ferran: Thank you. Yeah. I mean, I'm just at the stage of starting what might be a 
book on all of this, so is great to have that response and definitely pattern language, pattern 
generation started. Then you even get references to generative poetry in her older campus in 
in 1955 56 and talking about vectors. So and the thing is, if you look at it, you have to look at 
a lot of concrete poets. It's a bit like what was being talked about earlier. Um, if there's a 
collective group of people and they're working relationally, then they're building off each 
other's work. So because they all started using a particular type of typography, it's almost like 
there's a collapse. It's a post authorial approach. So it's decentering away from the self into 
like an auto poetics. So there's this collective mutation happening with one thing referencing 
another. And because it happens at a micro linguistic level, someone might just change one 


letter. You get exactly what's going on in text generation now, except that it's also embodied, 
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that there's a personal relationship, but a decentering and depersonalized format. And that's a 
critical kind of ground of emergence, out of which the limitations that we've now. Noam 
Chomsky is a key person because he was saying that, you know, the lack of meaning in 
ChatGPT is the problem. 


Bronnac Ferran: You're saying exactly the same thing in 1961. I've just been reading one of 
his things about linguistics. He actually was writing that an automated text is like a parrot, 
you know, and this is back in the early 1960s, and now he's still alive. And he's now saying, 
look, there's a lack of meaning here. And that, I think, goes back to what someone was saying 
earlier about negation, because a lot of these poems were they grew out of the post war, 
negation of language and a kind of an emptying out. So you get large blank spaces and a kind 
of specialist approach. And, you know, punctuation sometimes is left out. And that capacity 
to transfer a sort of an absence or a negation is what's not happening at the moment in 
generative texts, because it's just a pattern predictability rather than a unique directionality, 
which is what was being written about in the mid 60s. So we have a lot to learn from going 
back and remembering. And maybe that's what will happen when generative texts or AI starts 


being embodied, because then we'll get into all these issues again I think. 


Speaker3: Thank you. 


Frode Hegland: Oh, look, we're sitting in the chair. 


Speaker3: Oh. So this. 


Fabien Benetou: Can I show you something? Yeah. Yeah, yeah. Show it to me. Oh, my. So it's 
a mac, right? Might need some help. How do I command tab? 


Frode Hegland: Just like we stole from windows. 


Fabien Benetou: I don't use windows. 


Frode Hegland: So you need a URL. Right? 


Fabien Benetou: Yes. 
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Frode Hegland: And then command L or whatever. For location. And then type in your. 


Fabien Benetou: And I'm going to show some nonsense too. Oh, yes. Okay. 


Frode Hegland: Okay. Can be written on the 4 or 4 page. 


Fabien Benetou: Uh. So I have to show it because if I describe it, it's not going to help. I can't 
describe it properly. Uh, okay. So that's a kind of thing I'm working on, which is a 3D space. 
And you can, of course. I mean, of course you can put it in VR and in the headset. Why do I 
show it to you now is because all the text now, although it appears flat and all the letters 
individually, can be grabbed and then manipulated. I never, ever considered it for poetry. And 
when I see what you show, well, now I am. I don't know if it's a solution for this if it creates 
more problems than whatnot, but so this is of course, flat. I mean, you can see an animation 


on top. 


Bronnac Ferran: So you say you can move it as in with a cursor or something, but could you 


actually see it. 


Fabien Benetou: With our hands so you have the headset on like on I have here. Yeah. And 
then you, you see the text, you put your hand in front of it and then you grab and you rotate. 


You move either the whole sentence word or individual letters. 


Bronnac Ferran: Yes. Excellent. 


Fabien Benetou: So yeah, maybe that's a thing that could be used for exploration. 


Speaker3: Definitely. Yeah. 


Frode Hegland: So that was a really worthwhile callback to that work, because we've talked a 
lot about today about not wanting keyboards. Keyboards are great. Of course, if you have the 
basic training for a lot of stuff, but not for all stuff. So, you know, texts, you know, every 
once in a while someone will remind us that text and textile at the same route and all of that 


stuff. So if we do develop these gestures, we can start weaving whole sentences. We can do a 
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lot of stuff. Yeah. 


Bronnac Ferran: And texture is another word as well. 


Frode Hegland: Yes, absolutely. 


Speaker3: Okay. 


Frode Hegland: Thank you very much. 
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Glossary 


-liquid the document format for Author, which is a macOS wrapper with JOSN and RTFD. This is a 


free and open standard optimised for VR. 


3Dconnexion’s SpaceMouse® Wireless A 3-D input controller that looks like a rubberized hocky 
puck which can detect X, Y, and Z and Roll, Pitch, and Yaw deflections; it is more of a force vector 


input controller rather than an absolute positioning device. 


Abstract Data Space A visualization that takes n-dimensional objects (which is to say objects with 
an arbitrary number of attributes) and plots them with some arbitrary mapping function in two or three 


dimensions so they can be compared and clustered based on attributes of interest. 


academic someone who reads academic documents, usually associated with an academic institution, 


but not necessarily. 


ACM From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based 
international learned society for computing. It was founded in 1947 and is the world's largest 


scientific and educational computing society.” 
https://en. wikipedia.org/wiki/Association_for Computing Machinery 


ACM’s ‘Hypertext’ Conference piloted Visual-Meta in 2021. 


Adam Wern Working on spatial hypertext software. 


Future Text Lab contributor. 


Addressability refers to how something can refer to something else. In the digital world this 
primarily means in a local file structure and a server based network. In academia is can refer to 
citation or references. In the physical world it can refer to a formal location layout, such as a flat so 


and so in house so and so, as well as cordinates. It can also man relative addressing, such as saying 


take a left after the third yellow house. 


Addressability refers to how something can refer to something else. In the digital world this 
primarily means in a local file structure and a server based network. In academia is can refer to 
citation or references. In the physical world it can refer to a formal location layout, such as a flat so 
and so in house so and so, as well as co-ordinates. It can also man relative addressing, such as saying 


take a left after the third yellow house. 


Administrative Appended characteristics such as storage, preservation, access, including technical 


metadata and rights metadata 


AGI (Artificial General Intelligence). 


Al stage 2. 


If we are to see a world like the movies with artificial intelligence, AGI will need to come into full 
fruition. AGI would mean a machine would be capable of understanding the world as well as any 
human. All while maintaining the same capacity to learn how to carry out a huge range of tasks. The 
reality? AGI is not yet among us. It is a hypothetical form of AI. The potential is there but some 
believe it to be decades away! 

According to a study by Muller and Vincent C., the top-cited 100 AI researchers have a collective 
90% confidence that AGI will be among us by 2070. To determine if a machine is truly AGI, it has to 


pass the tests: 


Turing Test 


In this test, 2 humans and a machine look at something they haven’t seen and converse about it. The 
conversation is recorded and an evaluator determines which descriptor is the machine. The machine is 


only disqualified if the evaluator is positive it can identify it. (check out Touring Test on Wikipedia ) 


The Coffee Test (Wozniak) 


If AGI machines need to understand the world as well as humans do, they must know how to make a 


good coffee. In this test an ai robot is required to go into an average home and figure out how to make 
coffee. It involves finding the coffee machine and adding coffee, as well as sourcing a mug and 


correctly brewing the hot drink. 


Robot College Student Test 


A literal test! the machine would take the same university classes as humans and complete the exam. 
Some may argue that the machine is cheating with its built-in search engines but who is going to 


argue with a robot... 


Employment Test 


This is where we should begin to worry about an I-Robot world. This test determines whether a 


machine can perform at least as well as humans in the same jobs! 


AI ‘Artificial intelligence’ is “intelligence demonstrated by machines, as opposed to the natural 
intelligence displayed by animals including humans.” https://en. wikipedia. org/wiki/ 


Artificial_intelligence 


#Technology | Timeline: 20th century- 


Alan Kay is an American computer scientist best known for his pioneering work on object-oriented 


programming and windowing graphical user interface (GUI) design. 
Contributor to The Future of Text 


https://en.wikipedia.org/wiki/Alan_Kay 


Alan Laidlaw Independent researcher working on pixel-free interactions and peripheral interfaces. 


He looks at information technology the way a geomorphologist looks at civil engineering. 
Future Text Lab contributor. 


https://twitter.com/alanlaidlaw 


Alexander Burdiyan mintter.com 


Alexander Mehler Professor at the Goethe-Universitat Frankfurt at Goethe University Frankfurt, 


Computer Science where he runs the TTLab. 


Alexandr Burdiyan mintter.com 


ANI (Artificial Narrow Intelligence) 


Al Stage 1. 


Often called Weak AI, this is the AI already embedded in our daily lives. A nice way to describe it is 
that ANI is super-human at specific singular tasks. Like teaching an AI machine to play chess! and it 
will master it beyond human potential with incredible consistency. Challenge it in draughts though? 
and you’ll get a much needed ego boost. This machine can only play chess. Without prior training and 
programming in draughts, it doesn’t stand a chance. Think of AI as hundreds of interns capable of 


highly specific tasks more efficiently than humans. 


ANI has already transformed the way we live, but it has its limitations. A lot of you will already 
interact with ANI on a daily basis. Think of talking with Siri or Alexa on your smart devices, or facial 


recognition when opening your phone! 


Annie Murphy Paul From an interview in GQ: “We make better use of our cognitive resources, says 
Paul, when we use them in conjunction with “extra-neural” resources: our body (embodied cognition), 
our environment (situated cognition), and the people around us (distributed cognition). The brain 
evolved to move the body, to navigate through space, to interact with other people,” says Paul. 

“Those are these human strengths that we're totally putting aside when we focus on the brain and we 
think, “To get real thinking and real work done, I have to sit still, not talk to anybody, and just push 
my brain harder and harder.’ It's just not working very well.” 
https://www.gq.com/story/extended-mind-annie-murphy-paul 


Author of The Extended Mind. 


Annual Symposium hosted annually by Frode Hegland. 


Frode Hegland sees this as level C of Doug Engelbart’s Three Levels of Activity. 


Annual Symposium hosted annually by Frode Hegland. Part of The Future of Text. 


Frode Hegland sees this as level C of Doug Engelbart’s Three Levels of Activity. 


Ant Man A Marvel Super Hero capable of shrinking or growing the size of his body. 


Antonio Sena Antonio Sena is a bibliophile and ‘mediaphile' researching primary and secondary 
sources for near every subject since 1967. He holds a Ph.D. in Drawing (Drawing and Photography - 
Science and Art), Aggregate Professor at the Fine Arts University in Lisbon, 1993, where he taught 
Aesthetics (Iconology, Phenomenology and Semiotics), 1982-1994. 

He has a "no-Ph.D." in "Historia da Imagem Fotografica em Portugal 1839-1997”, where “I used a 
small relational database of my own, a Ph.D not argued voluntarily by me and published as a book, in 
1998, "Historia da Imagem Fotografica em Portugal 1839-1997" (in Portuguese). The graphic design 
of the book, also of my own, was done according to the intermediatheca concept. Since 1998, I am a 


permaculture farmer in Pico island, Azores archipelago.” 


Appended is any metadata which was added at the point of exporting/publishing the document or 


during authoring with the purpose of providing metadata on export. 


Apple The Computing, Media, and Cell Phone company previoulsy known as Apple Computer, Inc. 


Produces operating systems for macOS, 10S, watchOS, tvOS and visionOS. 


AR ‘Augmented Reality’ is an interactive experience that combines the real world and computer- 
generated content. The content can span multiple sensory modalities, including visual, auditory, 


haptic, somatosensory and olfactory. (Wikipedia). 


Included in the term XR. 


ARC Engelbart Concept: Augmentation Research Center, The name of Doug's lab at SRI where he 
proposed a system called H-LAM/T in 1962 and developed and in 1968 demonstrated NLS: oNLine 
System, his platform for shared knowledge work research, later renamed Augment and from which I 


decided on the name Author, since Author and Augment share etymological roots. 


Article A published paper, in a Journal, for academic workflows. 


ASI (Artificial Super Intelligence) 
Stage 3 AI. 


Just when you think AI has reached its limit, we bring you the AI of all AI. Super-Intelligence is quite 
self-explanatory. Oxford Philosopher Nick Bostrom describes ASI as “any intellect that greatly 
exceeds the cognitive performance of humans in virtually all domains of interest”. Again, this is a 


hypothetical agent and one which is decades away. 


ASI has huge potential but it also has huge risk. Stephen Hawkings feared its development could be 
the worst event in the history of our civilisation. If we create superintelligent machines we may put 


our own existence at risk, but this is just the plot for another sci-fi film. 


The reality is that AI is a tool for humans to better our existence. Advancements in artificial 
intelligence mean the sky’s the limit! The question is — will the limit bring more positives or 


negatives? We must tread carefully with AI whilst simultaneously embracing it. 


Augmented Text Company The company which produces these tools. 


https://www.augmentedtext.info 


Author a word processor from the Augmented Text company. 


https://www.augmentedtext.info/reader 


Author means in the context of this work someone who writes a document, which includes academic 
papers and books. 

Etymology: “mid-14c., auctor, autour, autor “father, creator, one who brings about, one who makes or 
creates” someone or something, from Old French auctor, acteor “author, originator, creator, instigator” 
(12c., Modern French auteur) and directly from Latin auctor “promoter, producer, father, progenitor; 
builder, founder; trustworthy writer, authority; historian; performer, doer; responsible person, 
teacher,” literally “one who causes to grow,” agent noun from auctus, past participle of augere “to 
increase,” from PIE root *aug- (1) “to increase.””’ From https://www.etymonline.com/search? 
q=Author 

Category: Software 

Timeline: - 


Can also refer to the Author software by The Augmented Text Company. 


Auto-Stereoscopic A display that sends a unique image to each eye without one’s having to wear 


glasses to achieve the 3-D effect. 


Avatar A representation of a real person, thing, or abstract concept that might appear as a high fidelity 
representaiton like a photorealistic Computer Generated Image (CGI) model of a famous person or a 


simplified stand-in like Microsoft’s Clippy <https://en.wikipedia.org/wiki/Office_Assistant>. 


Barbara Tversky Guest presenter at the Future Text Lab. 
Issue 1.1 on the 21st of January 2022. 


Professor emerita of psychology at Stanford University and a professor of psychology and education 


at Teachers College, Columbia University.Tversky specialises in cognitive psychology. 


Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019. 


Benediktine Cyberspace A 3-D visualization inspired by Michael Benedikt’s seminal text, 


“Cyberspace: First Steps”. 


A 3-D visualization of an N-Dimensional Abstract Data Space in which the X, Y, and Z axes can be 


associated with arbitrary data attributes such that a given point respresenting 3 of the N dimensions 


can be “unfolded” to access a linked visualization of 3 more dimensions and so forth. 


BibTeX is a specific format for conveying citation information within the LaTex environment, 
developed by Oren Patashnik and Leslie Lamport, released in 1985. The benefit of the system was to 
separate citation information from presentation style and it is human readable, though it slightly looks 
like code. It inspired the format of Visual-Meta and Visual-Meta contains a straight BibTeX section to 


allow the document which contains it to be cited. 


Bob Stein is the director of the Institute for the Future of the Book 

He founded The Voyager Company in 1985, the first commercial multimedia CD-ROM publisher, and 
The Criterion Collection in 1984, a collection of definitive films on digital media with in-depth 
background information (including the first films with recorded audio commentary). Worked with 
Alan Kay at the Atari Research Group on various electronic publishing projects. Currently works on 
The Tapestry Project. 

Timeline: 20 April 1946-. Born: NY, USA. Lives: NY, USA. 


Book a published work, on any substrate, inclduing paper or digital. 


Generally longer than a ‘paper’ but in terms of digital work, there is little technical difference, the 


difference is mostly social. 


Bootstrapping Doug Engelbart Notion.: The act of co-evolving the tool and human systems to make 
better tools and systems, thus pulling us up by our bootstraps. (For an overview, see the Bootstrap 


Paradigm Map). This is done on three levels of activity. 


Brandel Zachernuk AR / VR Creative technologist by day, digitally-embodied cognition enthusiast 
by night! Works at Apple. 


https://twitter.com/zachernuk 
https://www.youtube.com/channel/UCOP9cT-IwB6oxAcfvS1x_ ZQ 


https://futuretextlab.info/tag/bybrandel/ 


http://www.zachernuk.com/ 


“Something that I've been very passionate about trying to investigate and play with over the last 10 
years or so is what is the most pedestrian thing that you can do with virtual reality and a technology 
that was was word processing was reading and reading and thinking about what are the basic building 
blocks of that process of writing and reading that can fundamentally be changed by buying virtual 
reality? Realising that if you don't have a screen, you have the ability for information to mean what it 
means for your purposes, rather than the technical limitations that apply as a consequence of a mouse 
or keyboard or things like that, and also deeply invested in understanding some of the emerging 
cognitive science and and neurophysiological sort of views about what it is that the mind is and the 
way that we work best. 

So reading about learning about what people call for in cognition, it's embedded, embodied in active 
and extended mind and how that might pertain to what we should be doing with software and systems, 
both as well as hardware if necessary to make it so that we can. We can think properly and express 
properly and stuff.” 


https://youtu.be/2NcSCOrVw24?t=879 


Bronnac Ferran Birkbeck, University of London - Department of English. 


Bruce Horn created the original Macintosh Finder and the Macintosh Resource Manager for Apple 
Computer. His signature is amongst those molded to the case of the Macintosh 128K. He is a 


distinguished engineer at Siri and Language Technologies at Apple since June 2022. 


https://en.wikipedia.org/wiki/Bruce_Horn 


Canada https://en.wikipedia.org/wiki/Canada 


ChatGPT (Chat Generative Pre-trained Transformer) is a large language model-based chatbot 
developed by OpenAI and launched on November 30, 2022, that enables users to refine and steer a 
conversation towards a desired length, format, style, level of detail, and language. Successive prompts 


and replies, known as prompt engineering, are considered at each conversation stage as a context. 


https://en. wikipedia. org/wiki/ChatGPT (Nov 30 20203) 


Christopher Gutteridge is also a contributor to the first volume of The Future of Text. 
Lives: Southampton, UK. 


Works at the University of Southampton. 


Citations In this paper, the term’ citation’ is both the ‘citation’ mention in the body of the document, 
and its corresponding ‘references’ entry for the source material in the Reference Appendix at the end 
of the document, though for the sake of clarity I will mainly use ‘citation’ to refer to the in-body half 


and ‘reference’ for the entry of the source in the References section. 


Cognition “is the mental action or process of acquiring knowledge and understanding through 


thought, experience, and the senses”. 


https://en.wikipedia.org/wiki/Cognition 


Computational Text Vint Cerf’s notion of text which can be interacted with or changed based on 


inputs. Relates to the work of Bret Victor in particular. 


concept, defined concept, defined concepts means, in this context, a useful unit of thought or 


knowledge, which can be defined in Author, hopefully soon in Reader. Stored as text. 

The Map view uses the definition to draw lines where text from a definition is also present on the 
Map. 

When a docment is exported to PDF the Defined Concepts become Glossary Terms. 

This is as opposed to inferred concept. 


Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and headings). 


Cyberspace “The term "cyberspace" first appeared in the visual arts in the late 1960s, when Danish 


artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) constituted 


themselves as Atelier Cyberspace.” Used in this context to mean a digital world, not necessarily in VR 


or AR. 


Dame Wendy Hall, Wendy Hall is a British computer scientist. She is Regius Professor of Computer 


Science at the University of Southampton. 


Frode Hegland’s PhD primary supervisor, along with Les Carr and David Millard. 


Daveed Benjamin CEO Bridgit.io Author of first-of-its-kind augmented reality book, Pacha’s 
Pajamas: A Story Written By Nature that features Mos Def, Talib Kweli, and Cheech Marin 


David G Lebow A thought leader in the development of multi-document sensemaking technology as 


a service. 


https://www.hylighter.com 


David Johnson Co-author of Law and Borders, The Rise of Law in Cyberspace 


David Millard Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and Les Carr. 


David Small A researcher who envisioned ways to visualize text in VR during his time at MIT. 


Deena Larsen is a new media and hypertext author involved in the creative electronic writing 


community since the 1980s. From USA. 


https://en.wikipedia.org/wiki/Deena_Larsen 


Definition when it comes to glossaries, it is the user’s stated meaning of a term. 


Dene Grigar is a co-curator of The Future of Text Symposium. 


Digital artist and scholar based in Vancouver, Washington. Professor and Director of the Creative 


Media & Digital Culture Program at Washington State University Vancouver, USA. A prioneer of 
digital literature. Former president of the Electronic Literature Organisation. Curating and editing the 


NeXt. 


Timeline: -. Interests: Electronic Literature. 


Descriptive author, title, publisher, etc. and subject 


Document Can be any type of encapsulated information on a computer. 


DOI Document Object Identifiers. An effort to make addressing academic documents via the web 
more robust. 
Used in Author to let the user paste a DOI to cite an academic document which is then sent to 


CrossRef to be parsed into BibTeX which is then used to create a full citation. 


Doug Engelbart, Douglas Engelbart From Wikipedia: “He was an engineer and inventor, and an 
early computer and Internet pioneer. He is best known for his work on founding the field of human— 
computer interaction, particularly while at his Augmentation Research Center Lab in SRI 
International, which resulted in creation of the computer mouse, and the development of hypertext, 
networked computers, and precursors to graphical user interfaces. These were demonstrated at The 
Mother of All Demos in 1968. Engelbart's law, the observation that the intrinsic rate of human 
performance is exponential, is named after him.” 

He was also my mentor and greatly influened my work, resulting in my company called The 
Augmented Text Compant and my word processor being called Author, in honour of his ‘Augment’ 


system. Visual-Meta is inspired by his Open Hyperdocument work. 


Egypt https://en.wikipedia.org/wiki/Egypt 


Electronic Literature “Electronic literature or digital literature is a genre of literature encompassing 
works created exclusively on and for digital devices, such as computers, tablets, and mobile phones. A 


work of electronic literature can be defined as "a construction whose literary aesthetics emerge from 


wom 


computation", "work that could only exist in the space for which it was developed/written/coded—the 


digital space”.” https://en. wikipedia. org/wiki/Electronic_literature 


Eric Vicenti mintter.com 


ESA (European Space Agency) is an intergovernmental organisation of 22 member states dedicated to 


the exploration of space. 


https://en. wikipedia.org/wiki/European Space Agency 


Fabien Benetou Prototyping — European Parliament Innovation lab WebXR consultant — Former 


UNICEF Innovation Fund WebXR technical advisor. 
Future Text Lab contributor. 


fabien.benetou.fr 


Filter A filter can transform elements in a data set or exclude them from future processing. 


Find View in Author and in Reader (with Visual-Meta) where the user can see only the senteces with 


keyword. 


flatland a semi-humorous term we use to refer to traditional displays, as opposed to the augmented 
environments of VR and AR. We feel that it is important to be able to move data, including metadata 


between these environments. 


Folded View in Author and in Reader (with Visual-Meta) where the user can fold the document into 


an outline. 


Folding Folding of a document into a table of contents is enabled through Visual-Meta. 


Frame and Dictate The use of a gesture to indicate where to position a virtual screen to be 


immediately used with a speech recognition system to rapidly enter text in VR. 


Frame and Flatten The use of a gesture to select and re-orient a text-bearing surface in the real world 


to turn it into a manipulable text object in VR. 


Frode Hegland A PhD student at the University of Southampton at the WAIS group. 
Supervisors Wendy Hall, Les Carr and David Millard. 

Influenced by Doug Engelbart and Ted Nelson. 

I run an independent software development company called The Augmented Text Company. 
Editor of ‘The Future of Text’ series of books: https://futuretextpublishing.com 

Organizer of ‘The Future of Text’ annual symposia, founded 2011. 


Is a fan of the musical Hamilton. Lives in London. (the previous sentence is often used to demonstrate 
how the Map view works since there will be a line from me to Hamilton, since I have a relationship 
with Hamilton, but there is no line from Hamilton to me since Hamilton is not ‘aware’ of a 


relationship with me. 


Designed Visual-Meta. 


Future Text Lab A lab dedicated to studying how working with knowledge, particularly text, can be 
done effectively in VR/AR/Metaverse. We host Guest presenters and publish a Journal which is part 


of The Future of Text series. 


https://futuretextlab.info 


Gems software developed by Lorenzo Bernaschina. 


https://gemsnotes.app/ 


Geospatial Global Coordinates Latitude and Longitude on the Earth 


Germany https://en.wikipedia.org/wiki/Germany 


Gesture Interaction through hands and body. 


Giuseppe Abrami TTLab scientific assistant. 


glossary A glossary in this context means a local definition, by the author or editor of the document. 


Glossary means, in the context of this work, terms and definitions in a ‘Glossary’ Appendix in a 


document, as created by the author. 

This can be created in Author as a Defined Term which is then exported as a Glossary and included in 
Visual-Meta. 

Reader can parse the glossary terms in the Visual-Meta and make them interactive. 


This is different from a dictionary since dictionary definitions have general validity and Glossary 


terms only aim to express the author’s point of view. 
Defined Concept becomes Glossary Term when exported from Author to PDF. 


Inspired by discussions with Doug Engelbart. 


Glossary In the context of my work and thinking, a glossary is a set of Defined Concepts which an 
author has created and which is then exported as a Glossary for the reader. 

The primary purpose of this is for the author to have to think through their writing by explicitly 
stating what something is. The definition of the defined concept can then include text which also has a 
definition and in the Map view in Author this connection can be shown as a line when either defined 
term is selected. 

The secondary purpose is for the author to edit this Glossary to make sure it is coherent for export and 


it can then help a reader understand the author’s intentions. 


Glossing A way of elucidating parts of text. 


https://www.etymonline.com/search?q=gloss 


Goethe-Universitat Frankfurt https://www.goethe-university-frankfurt.de/en 


Google “is an American multinational technology company that focuses on search engine technology, 
online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial 
intelligence, and consumer electronics.” 

https://en. wikipedia. org/wiki/Google 


Timeline: 1998- . Location: USA 


GPS A Global Positioning System relying on the timing of the reception of pulses from a set of 


orbiting satellites to compute one’s Latitude and Longitude. 


GPT “Generative Pre-trained Transformer” inclduinig GPT-2 and GPT-3. 
https://en. wikipedia. org/wiki/GPT-2 

https://en. wikipedia. org/wiki/GPT-3 

Product of OpenAI. 


GPT-3 ‘Generative Pre-trained Transformer 3’ (GPT-3; stylized GPT-3) is an autoregressive language 


model that uses deep learning to produce human-like text. https://en. wikipedia. org/wiki/GPT-3 


“In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate 


wow 


computer code, poetry, and prose is not just "amazing", "spooky", and "humbling", but also "more 
than a little terrifying". https://www.worldcat.org/issn/0362-433 1 
A product of OpenAI. 


Timeline: 11 June 2020-. Keywords: Deep Learning. AI. ML. Machine Learning. 


Granularity of Concern The level of detail or scale at which one wants to work. 


Hamilton a father of the US and a musical by Lin-Manuel Miranda which has influenced Frode 


Hegland greatly. The real person is also an inspiration since he used the power of text to write a new 


society into existence. 


Frode Hegland uses Hamilton in presenting the Map and Defined Concepts in Author since it’s a fun 


and easy way to present the one-way relationship. 


Hand Tracking Technology that uses cameras or other sensors to determine the position and 


orientation of a user’s hands and fingers enabling it to recognize hand gestures. 


Handoff An apple technology to cut and paste and share state (i.e. current preferences, selections, or 
anything else that might change from a program’s intitial configuration) across Apple Devices 


associated with the same user account. 


Heads-Up-Display A translucent display that you can read while simultaneously seeing whatever is 


in front of you beyond the display. A framed form of AR, as used in aircraft and to some extent cars. 


Headset A more or less sleek visor containing accelerometers, gyroscopic orientation sensors, 
optional pass through video cameras and LIDAR sensors in conjunction with a high resolution flat 
pannel display unit using optics to provide a stereoscopic display of a virtual, augmented, or real 


world. 


HMD ‘Head Mounted Display’ for use in VR or AR. 


Horacio Herrera mintter.com 


Howard Rheingold is a critic, writer, and teacher, known for his specialties on the cultural, social and 
political implications of modern communication media such as the Internet, mobile telephony and 
virtual communities (a term he is credited with inventing). Born in Arizona USA. 

Friend of Doug Engelbart. Inspiration and friend to Frode Hegland, who discovered Doug from his 


“Tools for Thought’ book. 


Presented at The Future of Text Symposium. 


https://en. wikipedia.org/wiki/Howard_ Rheingold 


HTML “The HyperText Markup Language or HTML is the standard markup language for documents 
designed to be displayed in a web browser.” https://en.wikipedia.org/wiki/HTML 


A technology we are also using, along with PDF. 


Human-Computer Interaction The field of study exploring user interfaces to computing systems; 
the premier conference series in this field is the ACM CHI (Computer-Human Interaction) conference 


series. 


Hypercard A 3-D graphic depicting a physical business card bound to an arbitrary chunk of data in 
the novel Snow Crash, not to be confused with the index card inspired End User Programming 


environment from Apple. 


hypertext a term invented by Ted Nelson for interactive and connected digital text. Today it is very 


much about clicking on links but in Ted’s early work it was all about freedom for the user. 


Hypertext a term invented by Ted Nelson for interactive and connected digital text. Today it is very 


much about clicking on links but in Ted’s early work it was all about freedom for the user. 


Immersed Immersed is a virtual reality product, working productivity software for virtual offices. 


https://immersed.com 


Immersive Immersive experieces create the illusion of being part of the experience. 


INFRASTRUCTURE The systems of interactions. 


Intelligence Augmentation Doug Engelbart presented the notion of augmenting human intellect in 


his seminal 1962 paper, which would be the basis for his ‘mother of all demos’ in 1968: 

“We need to improve how we augment a group’s (small, large, internal, global etc.) capability to 
approach urgent, complex problems to gain more rapid and better comprehension (which can be 
defined as more thorough and more critical) which result in speedier and better solutions (more 
contextual, longer lasting, cheaper, more equitable etc.). And furthermore, we must improve our 
improvement process (as individuals and groups).” 


Doug Engelbart (Engelbart, 1962) 


Interactions How we can interact with information, and eachother, through computer systems. 


Intrinsic Attribute of a data object that are not represented by that object’s spatial position in a given 
visualization but which may be represented by its size, shape, color, transparence, or other 


positionally independent visualization technique. 


IPFS The Interplanetary File System is a content addressable global file storage and retrieval network 
based on cryptographic hashes of their stored contents rather than the network address at which those 


files are located. 


Ismail Serageldin is a friend and supporter of Frode Hegland. They first met at the Science & 
Technology Forum in Japan quite a few years ago. 


He is Egyptian, Founding Director of the Bibliotheca Alexandrina (BA) and was Vice President of the 
World Bank. 


Italy A country 


Jacob Hazelgrove Programmer for all Augmented Text tools for Frode Hegland, including Author 
and Reader, as well as imlementor of Visual-Meta export from Author and import and interaction in 


Reader. 


James Addison Works on Reducing the friction between ideation & creation. 
https://twitter.com/JungleSilicon 


https://t.co/USgyvE]jY3 


James O’Sullivan Senior Lecturer in Digital Humanities at University College Cork. He is the PI 
(Ireland) for the C21 Editions project. His recent publications include The Bloomsbury Handbook to 


the Digital Humanities (Bloomsbury 2023). jamesosullivan.org 


Jaron Lanier “American computer scientist, visual artist, computer philosophy writer, technologist, 
futurist, and composer of contemporary classical music. 

Considered a founder of the field of virtual reality.” https://en.wikipedia.org/wiki/Jaron_Lanier 

VR pioneer. 


Timeline: 3 May 1960- . Born: NY USA 


Jasper Hustedt Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


John Bowditch Throughout his long and distinguished career,Mr. Bowditch has worked extensively 
on the material culture of late 19th century and early 20th century industrial infrastructure specializing 
in the activation of historical machinery in a museum setting, including machine shops and steam 
technology. Included among his many important hands on restoration projects while Curator of 
Industry at the Henry Ford Museum and Greenfield Village was re-tooling the many shops at the 
historical museum village to run on a series of leather belts, an achievement he duplicated while a 
consultant at the Thomas Edison National Park. This was only one of his many consultations with 
museums including the Smithsonian, Kregel Windmill Factory Museum, Illinois and Minnesota 
Historical Societies, and the Museum of Science and Industry in Chicago. Mr. Bowditch is also a well 


known expert on early gas and electrical lighting and the early years of audio recording. 


Jonathan Finn is a developer of creativity apps (e.g. Sibelius music writing system). 


Contributor to The Future of Text volume 2 and editing for volume 3. 


Journal The monthly Journal of the Future Text Lab. Inspired by Doug Engelbart’s journal and 
academic journals in general. Published as PDF with Visual-Meta and in .liquid and soon HTML on 


https://futuretextpublishing.com/journal/ 


Journal A collection of articles. For academic use, usually the record of the work of society, such as 


any of the many special interest groups at ACM. 


JSON (JavaScript Object Notation,) An open standard file format and data interchange format that 
uses human-readable text to store and transmit data objects consisting of attribute—value pairs and 


arrays (or other serializable values). 


https://en. wikipedia.org/wiki/JSON 


Julian Gagel Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


Julio Garcia mintter.com 


Jun Rekimoto A researcher at SONY. 


Kevin Bénisch Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


Lab I host an hour on Zoom every Monday at 4pm UK time for a community of people who are 
interested in The Future of Text. They have so far really helped polish Visual-Meta and provide 
thoughtful dialogue. 


Leon van Kammen Creative Technologist & Researcher. 


Future Text Lab contributor. 


https://github.com/coderofsalvation 


Leon Van Kammen Leon is a Dutch audiovisual prototyper and researcher currently based in 
Budapest, Hungary. 

He is currently working on XR Fragments, in which he strives to bridge his diverse experiences from 
the worlds of audiovisual productions, programming, and the web. 

Future Text Lab contributor. 


https://2wa.isvery.ninja https://www.linkedin.com/in/leonvankammen 


Les Carr, Leslie Carr Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and David 
Millard. 


LIDAR (L)ght (D)etection (A)nd (R)anging 


Light-Field A horizontal parallax 3-D display technology that simulates 3-D for mutliple viewers as 


they shift positions along a horizontal axis. 


Liquid is a text interaction tool from the Augmented Text company. It can be used in Author, Reader 


and most macOS applications. 


https://www.augmentedtext.info/liquid 


Liquid Information, liquid Frode Hegland’s philosophy of interactive computing. 


Developed while a student in New York, the fundamentals are removing barriers to rich interaction. 
Later influenced and expanded by the work of his mentor Doug Engelbart and fleshed out in 


discussions with Sarah Walton. 


https://www. liquidinformation.org 


Liquid Mode “Adobe Liquid Mode turns PDFs into a more readable format for smaller screens. You 


can rest assured that it doesn’t change the PDF itself, but rather, it reformats headings, paragraphs, 
and images to create a better flow for mobile viewing. Plus, it creates an outline with collapsible and 
expandable sections for you to easily navigate within larger documents. 

Liquid Mode also makes mobile PDFs more responsive to your needs. You can search text within the 
document, expand its images, and — to make it even easier on the eyes — alter the font size and 
spacing between words to suit your reading preference.” 
https://www.adobe.com/acrobat/hub/how-to/what-is-adobe-liquid-mode 


#technology 


LiSA Software by Frode Hegland, featuring the voice of Janine Earl. 


Discontinued: http://www. liquid.info/lisa/ 


Livia Polanyi is also a contributor to the first volume of The Future of Text. 


Lives: NY, USA. 


LLM ‘Large Language Model’ is a type of language model notable for its ability to achieve general- 
purpose language understanding and generation. 

LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during 
training and consuming large computational resources during their training and operation. 

LLMs are artificial neural networks (mainly transformers) and are (pre-)trained using self-supervised 
learning and semi-supervised learning. 

As autoregressive language models, they work by taking an input text and repeatedly predicting the 
next token or word. 

Up to 2020, fine tuning was the only way a model could be adapted to be able to accomplish specific 
tasks. Larger sized models, such as GPT-3, however, can be prompt-engineered to achieve similar 
results. They are thought to acquire embodied knowledge about syntax, semantics and "ontology" 
inherent in human language corpora, but also inaccuracies and biases present in the corpora. 

Notable examples include OpenAI's GPT models (e.g., GPT-3.5 and GPT-4, used in ChatGPT), 
Google's PaLM (used in Bard), and Meta's LLaMa, as well as BLOOM, Ernie 3.0 Titan, and 
Anthropic's Claude 2. 


https://en. wikipedia.org/wiki/Large language model 29 Nov 2023 


Luc P. Beaudoin Faculty of Education, Simon Fraser University 


Machine learning Machine learning (ML) is a field of inquiry devoted to understanding and building 


methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. 
A type of Artificial Intelligence (AI). 
https://en. wikipedia. org/wiki/Machine_learning 


#Technology. Timeline: 20th century-. 


manuscript the authoring format, such as Microsoft Word, which is then either shared as-is, and stays 


editable, or is exported to be published in a publish format, such as PDF. 


Manuscript the authoring format which is used when writing, such as Microsoft Word. 


It is then either shared as-is, and stays editable, or is exported to be published in a published 


‘Document’ format, such as PDF. 


map View in Author. 


Here the user can place text anywhere they want and any defined concepts will automatically appear 
here. If there are defined concepts on the map, the user can click on them and lines will emanate to 


any text on the map which is in that text’s definition. 


Mark Anderson Independent researcher in Hypertext and Knowledge systems. Associated with the 
Web & Internet Science (WAIS) Lab at Southampton University. 


Mark was heavily involved in the technical specification and ACM launch of Visual-Meta. 

Mark has a background in organisational structure, knowledge, process change and data interchange. 
is a co-curator of The Future of Text Symposium. 

PhD from the University of Southampton. 


Research interests includes hypertext and VR. From Southampton, UK 


https://www.southampton.ac.uk/publicpolicy/support-for-policymakers/blogs/evidence-to-policy- 


blog/mark-anderson-cabinet-office-placement.page 


Mark Bernstein Mark Bernstein is founder and Chief Scientist of Eastgate Systems. 


http://www.eastgate.com/people/Bernstein. html 


Massively Multiplayer Online Role Playing Game An interactive networked game hosting 


hundreds or thousands of players pursuing missions and commerce in an extended fictional universe. 


Matthias Miiller-Prove “Is a Computer scientist, interaction designer and the founder of the Chrono 
Research Lab. So doing lots of heritage stuff. Also sometimes refer to myself as a kind of software 


archaeologist. That was my introduction.” 


https://mprove.de 


Mesopotamia a historical region within the Tigris—Euphrates river system in the northern part of the 
Fertile Crescent where early writing developed (in parallel to, or influenced by/influencing Egyptian 


writing), including the use of colophons. 


Meta is a large social media company which is focusing on what they term the ‘metaverse’. 
Formerly called Facebook. 
https://about.facebook.com 


Timeline: 4 February 2004- 


metadata information about other information, in the case of documents, this can include structural 


information (headings for example), biblio 


METADATA is information about other information, in the case of documents, this can include 
structural information (headings for example), how they connect to other documents (References) and 


who created the document (self-citing) 


Metaverse From Wikipedia: “A metaverse is a network of 3D virtual worlds focused on social 


connection” 
“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash” 
https://en. wikipedia. org/wiki/Metaverse 


At the time of writing, 2022, the term has been popularised by Meta, the company previously called 
Facebook. Most often used in conjunction with AR/VR augmented environments, not such much the 


3D worlds where the user access the world through a flat screen. 


Mez Breeze While studying for a degree in Applied Social Science in the early 1990’s, she started 


using the Internet to create interactive digital works. 


Since then, Mez has published over 300 seminal works including award-winning electronic based 
writing, Virtual Reality literature, XR sculptures, Artificial Intelligence artworks and books, games, 
and other genre-defying output all while teaching/mentoring, archiving, and supporting digital art and 


electronic literature. 


https://mezbreeze.itch.io/portraits-volume-one https://www.mezbreezedesign.com 


Michael Bonfert Researcher, Digital Media Lab, University of Bremen 


www.uni-bremen.de/dmlab 


Michael Pidd Director of the Digital Humanities Institute, University of Sheffield. He is the PI (UK) 
for the C21 Editions project. 


Mini-Me An “Austin Powers” movie reference. 


Mixed/Extended XR A hybrid of the VR and AR in which the system maintains a 3-D model of its 
surroundings permitting it to retexture or hide actual objects or simulate the insertion of virtual 
objects into the real environemnt such that they obey the laws of physics and are potentially 


indistinguishable for actual objects they might depict. 


Mnemonic ID’s Names like Purple Elk that are readily committed to memory without having to be 
written down, which usually take the form of a short memorable gramatically valid phrase or 


description that can also obviate the need for a distinct password. 


Names All Names in a document. 


View in Author and in Reader (with Visual-Meta) to show all named entities (and headings). 


Navigable An experience in which one can change one’s viewpoint and/or orientation. 


Nayana Prakash DPhil Researcher, Oxford Internet Institute.» 


Presented at the 2023 Future of Text Sympsium. 


neural chip A direct brain implant that interfaces with brain cells by stimulating them electrically 
(e.g. a visual cortex interface that tricks the brain into thinkng it is receiving senory input from one’s 


eyes). 


NIC Engelbart Concept: Networked Improvement Community “Consider an "Improvement 
Community" (IC) as collectively engaged in improving an agreed-upon set either of individual 
capabilities, or of collective group capabilities-e.g. a professional society. Let's introduce a new 
category, a "Networked Improvement Community" (NIC): an IC that is consciously and effectively 
employing best-possible DKR (Dynamic Knowledge Repository) development and usage.” 


(augmenting society's collective IQ). 


NLS From Wikipedia: “NLS, or the "oN-Line System", was a revolutionary computer collaboration 
system developed in the 1960s. Designed by Douglas Engelbart and implemented by researchers at 
the Augmentation Research Center (ARC) at the Stanford Research Institute (SRI). The NLS system 
was the first to employ the practical use of hypertext links, the mouse, raster-scan video monitors, 


information organized by relevance, screen windowing, presentation programs, and other modern 


computing concepts. It was funded by ARPA (the predecessor to Defense Advanced Research Projects 
Agency), NASA, and the US Air Force.” https://en. wikipedia.org/wiki/NLS_(computer_system) 


SRI sold NLS to Tymshare in 1977 and renamed it Augment. 


noise canceling earphones Earphones that generate an inverse waveform to cancel out environment 


sounds. 


NYU (New York University) is a private research university in New York City. Chartered in 1831 by 
the New York State Legislature, NYU was founded by a group of New Yorkers led by then-Secretary 
of the Treasury Albert Gallatin. 


https://en. wikipedia.org/wiki/New_York_ University 


OpenAL is an artificial intelligence (AI) research laboratory. 

“_..consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAI 
Inc. 

The company, considered a competitor to DeepMind, conducts research in the field of AI with the 
stated goal of promoting and developing friendly AI in a way that benefits humanity as a whole. The 
organization was founded in San Francisco in late 2015 by Elon Musk, Sam Altman, and others, who 
collectively pledged US$1 billion. Musk resigned from the board in February 2018 but remained a 
donor. In 2019, OpenAI LP received a US$1 billion investment from Microsoft.” 

https://en. wikipedia.org/wiki/OpenAI 

https://openai.com 

Products include: GPT-3 and DALL-E. 

Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and John 
Schulman. 


Timeline: 11 December 2015-. 


OpenAL is an artificial intelligence (AI) research laboratory consisting of the for-profit corporation 


OpenAI LP and its parent company, the non-profit OpenAI Inc. 


Products: GPT-1. GPT-2. GPT-3 


https://openai.com 


https://en. wikipedia.org/wiki/OpenAI 


Orientation Roll, Pitch, and Yaw values. 


Paper is a general term for a student or academic document in general. 
Primarily in PDF when published or handed in. 


In manuscipt/editable/personal form it is generall in the Microsoft Word format. 


pass-through video When wearing a headset that prevents you from seeing the room around you, 
pass-through video relies on cameras aligned with your left and right eyes to feed a video image of 
what that eye would see if one was not wearing the display to that eye to create the illusion that you 


are looking at your souroundings without having the display on. 


Patrick Lichty Assistant Professor, Animation/Multimedia Design, College of Arts and Creative 


Enterprises, Zayed University, Abu Dhabi 


Patrick Lichty Assistant Professor, Animation/Multimedia Design, College of Arts and Creative 


Enterprises, Zayed University, Abu Dhabi 


PDA ‘Personal Digital Assistant’. This was before smartphones. 


PDF ‘Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print 
to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for 
users to interact with the document in useful ways, while staying compatible with the basic PDF 


format. 


It is an export format rather than a manuscript/working format. 


PDF ‘Portable Digital Format’ developed by Adobe, now free with no license restrictions. It is a print 
to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for 
users to interact with the document in useful ways, while staying compatibel with the basic PDF 


format. 


perspectives how you look 


Peter J. Wasilko Peter J. Wasilko, Esq. is an Attorney, Programmer, and Independent Scholar 
residing in Ossining, New York (one of the historic river towns overlooking the Hudson an hour North 


of New York City). 
Future Text Lab contributor. 
https://futuretextlab.info/2022/12/27/peter-j-wasilko-esq/ 


Timeline: -. Lives in: New York, New York, USA. 


Phil Gooch Founder & CEO Scholarcy. 


https://www.scholarcy.com 


Philipp Stephan Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


photo-realistic Indistinguishable from a photograph. 


Pick and Drop A mean of moving data between devices by detecting the presence of a “Pen” gadget, 
assocating the network addrees of the data with the ID of the “Pen” and then detecting when the 
“Pen” is tapped on a receiving device, where its ID is looked up over the network to retrieve a copy of 


the original data. 


Pluck and Plop A pair of gestures for transferring content across targets in VR. 


Principle of Severability The notion that people won’t invest in creating user generated content if 


they can’t get it out of the environment in which they create it. 


Principle of Transferability The notion that for text (and other content) to realize its full untility in 
VR it must be easy for users to fluidly import and export it and to transfer it between platfroms and 


virtual worlds. 


Purple Elk Humane / Mnemonic ID’s inpsired by Purple Numbers (Doug Engelbart’s fine grained 
hyperlinking approach) and Jeff Raskin’s work. 


QR Code A 2-D variant of a Bar Code suitable for encoding a URL that is widely supported by cell 


phones, tablets, and laptops. 


Rachel Horst Faculty of Education, University of British Columbia 


Reader a PDF viewer from the Augmented Text company. 


https://www.augmentedtext.info/reader 


Reader a PDF viewer from the Augmented Text company. 


https://www.augmentedtext.info/reader 


Reader Software is a minimalist PDF viewer for macOS. 

Can read any PDF and can provide added interactions if the PDF has Visual-Meta attached. This can 
either be produced by Author or any other word processor with Visual-Meta capability, or downloaded 
from an online repository which features Visual-Meta, such as the ACM digital library. Produced by 
The Augmented Text Company LTD, with programming by Jacob Hazelgrove. 
https://www.augmentedtext.info 


#Software | Timeline: 12 July 2019- 


Real World The world as we experience it without any computer mediation. 


Reference Section is a section to list all the sources of the citations in the article or paper. 


References is a list of all the citations a document uses, in an Appendix. In-Body citation, point to 
these References. This language is not fixed, it is sometimes used interchangably with Bibliography 


but in my context a Bibliography is a list of work not expressly cited but which are relevant. 


in this context ‘Reference’ with uppercase ‘R’ refers to the appendix in an academic document which 
lists cites sources. In contrast, the citation in the body of the document is referred to as in-body 


citation. 


Rob Swigart American novelist, poet, short story writer, futurist, and archaeology scholar best known 
for his satirical work, archaeology writing, science fiction, and interactive novel computer game, 
Portal (Activision, 1986). He is the author of sixteen books, including fourteen novels, one business 


book, and one translated prose poem. 
Future Text Lab contributor. 


https://en. wikipedia.org/wiki/Rob_Swigart 


robust is defined as the metadata being available for as long as the document is readable, even if the 


reader software changes or the data format of the document changes. 


Rollo Carpenter Artificial Intelligence Developer of Cleverbot. 


https://en. wikipedia.org/wiki/Rollo_Carpenter 


Rome a city in Italy 


Russia or the Russian Federation, is a transcontinental country spanning Eastern Europe and Northern 


Asia. It is the largest country in the world. 


Sam Brooker is an academic specialising in digital communication. 
Associate Professor of Digital Communications at Richmond American University London, UK. 


His research explores the relationship between digital technologies & theories of literature and 
culture. He is a regular contributor to ACM Hypertext and Social Media, frequently serving on the 
programme committee. Additional contributions have featured at the ICIDS and SHARP conferences. 
I am a member of the PRCA International University Advisory Group and academic reviewer for 


numerous journals. 
https://www.richmond.ac.uk/school-of-communications-arts-social-sciences/dr-sam-brooker/ 


Timeline: -. Research interests: Digital cultures. Electronic literature. Book history. Digital 


humanities, Theories of authorship. Transmedia. Lives in: London, UK. 


Sam Winston Artist. His practice is concerned with language not only as a carrier of messages but 
also as a visual form in and of itself. Initially known for his typography and artist’s books he employs 
a variety of different approaches including drawing, performance and poetry. 


https://www.samwinston.com 


Scott Rettberg Researcher and writer teaching digital culture and electronic literature at the 


University of Bergen, Norway. 
https://twitter.com/scottrettberg 


Timeline: - 


Search Search in this context refers to instant search enabled by Liquid. 


Shaduan Babbili Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


Sophie Whittle Research Associate for the C21 Editions project at the Digital Humanities Institute, 


University of Sheffield. She recently completed her PhD in historical linguistics and is an organiser at 


Sheffield Feminist Archive in her spare time. 


sophiewhittle.co.uk. 


Spain https://en.wikipedia.org/wiki/Spain 


Spatial Computing “was defined in 2003 by Simon Greenwold, as “human interaction with a 
machine in which the machine retains and manipulates referents to real objects and spaces”. With the 
advent of consumer VR, AR and mixed reality, companies use ‘spatial computing’ in reference to the 
practice of using physical actions (head and body movements, gestures, speech) as inputs for 
interactive digital media systems, with perceived 3D physical space as the canvas for video, audio, 


and haptic outputs. It is also tied to the concept of ‘digital twins’. 
The terminology used by Apple for Vision Pro. 


https://en. wikipedia. org/wiki/Spatial_computing 


Speech to Text Analog Digital Conversion 


Structural inherent characteristics such as pages, headings etc. 


Student for my work, the student is the primary user of Author for writing and Reader for reading. 


Sub-Spaces This can be a sub-volume within a conataining volume or in the case of a Benediktine 


Cyberspace, a set of alternate dimensions contained within a selected point in its enclosing space. 


Tangible Citations The simulation of visible links between references in the 3-D space inspired by 


the side-by-side text visualizations envisioned by Ted Nelson’s Xanadu Project. 


Tangible Hypercards A passive form of Pick and Drop using cards with a QRCodes and Mnemonic 


ID’s to eliminate the need for custom hardware. 


Teacher in this context it the primary reader of a student paper. 


Ted Nelson, Theodor Holm Nelson coined the term ‘hypertext’. 


Text is the basic ’stuff’ of this work. 


The Construct A self-contained small scale virtual world variously depicted as an infinite white void 


or martial arts Dojo in The Matrix films. 


The Future of Text Annual Symposium and Book Series (first volume published 2020) as well as 
community for fostering dialogue around the future of text which I started over a decade ago and 


which is often co-hosted or presented by Vint Cerf. 


https://futuretextpublishing.com 


The Future of Text Annual Symposium and Book Series (first volume published 2020) as well as 
community for fostering dialogue around the future of text which I started over a decade ago and 


which is often co-hosted or presented by Vint Cerf. 


https://futuretextpublishing.com 


The Matrix A series of science fiction movies in which people are grown in pods with direct Neural 
Chip interfaces to a virtual world depicting America in the 1990’s that is also populated by Artificially 


Intelligent “Agents”. 


Theresa Berg Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


Time Index A reference to a point in time or rage of times within a time-varying dataset. 


Timo Liittig Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


tool for thought is both a book and a category of tools, primarily software tools, to augment how 


people think. 


The book: “Tools for Thought: The History and Future of Mind-Expanding Technology is a work of 
"retrospective futurism" in which Smart Mobs author Howard Rheingold looked at the history of 
computing and then attempted to predict what the networked world might look like in the mid-1990s. 
The book covers the groundbreaking work of thinkers like Alan Turing, John von Neumann, and 
J.C.R. Licklider, as well as Xerox PARC, Apple Computer, and Microsoft (when Microsoft was 
"aiming for the hundred-million-dollar category"). Rheingold wrote that the impetus behind Tools for 
Thought was to understand where "mind-amplifying technology" was going by understanding where 


it came from.” https://en.wikipedia.org/wiki/Tools_for_ Thought 


Touch Touching a device to interact. 


UK https://en.wikipedia.org/wiki/United_Kingdom 


Ukraine https://en.wikipedia.org/wiki/Ukraine 


#Location 


University of Southampton University in the UK. 


USA https://en.wikipedia.org/wiki/United_States 


Viewpoint X, Y, and Z coordinates in a 3-D space. 


Viewspec A data structure containing all of a visualization’s control parameters including referenced 


data sets needed to reproduce that visualization from a given perspective. 


ViewSpec Viewspecs — short for "view specifications”. Allows you to control your view of a 


document. 


https://www.dougengelbart.org/content/view/218/ 


ViewSpec Engelbart Notion: View Specifications for a user to see their work in different views to 


gain new perspectives and to navigate their document. 


Viewspec Reification Encapsulating a viewspec as a manipulable object. 


ViewSpecs Viewspecs — short for "view specifications" — allow you to control your view of a file. 


https://www.dougengelbart.org/content/view/2 18/ 


Vint Cerf, Vinton Gray Cerf Co-inventor of the Internet, VP and Chief Internet Evangelist for 
Google. Chairman of the Marconi Society. Former executive at MCI, the Corporation for National 
Research Initiatives, the Internet Corporation for Assigned Names and Numbers (ICANN), the 
American Registry for Internet Numbers (ARIN), the Association for Computing Machinery (ACM) 
and member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, AAAS, American 
Academy of Arts and Sciences, and American Philosophical Society. Member of the US National 
Academies of Engineering and Science and foreign member of the Royal Society and the Royal 
Swedish Engineering Society. 

He has been a strong and vocal supporter of Frode Hegland’s work, and Visual-Meta in particular. 
Category: Person 


Timeline: 23 June 1943 


Vision Pro XR headset. Made by Apple. Refered to as Spatial Computing. Runs visionOS. 


visionOS Apple’s XR operating system, first deployed on the Vision Pro. 


Visual-Meta is my approach for embedding metadata in documents, initially PDF, to allow viewer 
software to provide rich interactions. The Visual-Meta is added as an appendix to the end of the 


document in a format inspired by BibTex. 


http://visual-meta.info 


Visual-Meta An open and robust way to augment flat PDF documents to make them more interactive, 


by The Augmented Text Company people. 


http://visual-meta.info 


Visual-Meta A open and robust way to augment flat PDF documents to make them more interactive, 


by The Augmented Text Company people. 


http://visual-meta.info 


Voice Speaking and listening for interaction. 


VR ‘Virtual Reality’ is a simulated experience that employs pose tracking and 3D near-eye displays to 
give the user an immersive feel of a virtual world (Wikipedia). 
In contrast to AR where the user can see their physical environment. 


Included in the term XR. 


VR ‘Virtual Reality’ is a simulated experience that employs pose tracking and 3D near-eye displays to 


give the user an immersive feel of a virtual world (Wikipedia). 


Included in the term XR. 


Wendy Hall Frode Hegland’s PhD primary supervisor, along with Les Carr and David Millard. 


Xbox A popular high end video game system. 


XR ‘eXtended Reality’, a term including both Virtual Reality (VR) and Augmented Reality (AR). 
This is a useful term since headsets increasingly feature passthrough video, allowing the user to toggle 


between VR and AR modes. 


XR Fragments A mechanism for linking VR Experiences on the web so clicking on a virtual object 
can effect a perspective or scene change analogous to clicking a hypertext link to follow it on an 


ordinary web page. 


Yannick Heinrich Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University 


Frankfurt, Computer Science. 


Yohanna Joseph Waliya Enseignant et chercheur a l'université de Calabar. 


UNESCO Janusz Korezak Fellow, ELO Research Fellow, Winner of Janusz Korczak Prize for Global 
South 2020, Creator & Curator of MAELD & ADELD. Executive Director of AELA & ADELI. 
Auteur de La récolte de vie, Climatophosis (Winner Best Use of DH for Fun 2020) etc. 


https://africanelit.org/ 
https://orcid.org/0000-0003-4456-7458 
https://unical-ng.academia.edu/WALIYAYOHANNAJOSEPH 


https://www.researchgate.net/profile/Yohanna_Waliya 


Endnotes 


' Generated with the prompt “futuristic gold metal robot with many cables and swirls of light, reading 
a book with a grey cover against a white background, called 'The future of text’. Many other books 


floating nearby” with www.imagine.art using Imagine V4 

? https://webb.nasa.gov/content/science/firstLight.html 

3 Ts the universe twice as old as we thought? 
https://cosmosmagazine.com/space/astrophysics/universe-27-billion-years-old/ 


4 hittps://www.livescience.com/space/astronomy/james-webb-telescope-finds-an-extreme-glow- 


coming-from-90-of-the-universes-earliest-galaxies 
> As first proposed by Doug Engelbart. 
© https://youtu.be/GEPhLqwK 06g?si=tMm VtjOLLig31h35 


T https://www.dropbox.com/s/pnal fvfSsrbctwg/ 
xrOS%3A%20What%20to%20Expect%20From%20the%20Software%20Designed™%20for%20A pple 
%27s%20AR%3AVR%20Headset%20-%20MacRumors.pdf?dl=1 

8 What performance-enhancing stimulants mean for economic growth 

https://www.economist.com/ 
https://www.economist.com/finance-and-economics/2023/05/25/what-performance-enhancing- 
stimulants-mean-for-economic-growth 

° Prepare for the Textpocalypse By Matthew Kirschenbaum in The Atlantic. 
https://www.theatlantic.com/technology/archive/2023/03/ai-chatgpt-writing-language-models/ 
673318/?utm_source=apple_news 

10 https://speechify.com 

'l Apple ‘Live Text’ has gone some way to address this recently. 
https://support.apple.com/en-gb/HT2 12630 

'2 And some day, multiple high quiality projectors onto the world. 

'3 https://scholarlykitchen.sspnet.org/2023/04/13/textpocalypse-a-literary-scholar-eyes-the-grey-goo- 
of-ai/ 


'4 An article blogged and subsequently lost to hacking and spam. 


'S Douglas Engelbart 1963: A Conceptual Framework for the Augmentation of Man’s Intellect. 
https://dougengelbart.org/pubs/augment-133183-AHI-Vistas.html 


'6 Tvan Sutherland 1963: Sketchpad - A Man-Machine Graphical Communication System. https:// 
www.cl.cam.ac.uk/techreports/UCAM-CL-TR-574.pdf 


'7 Apple Lisa IN Matthias Miiller-Prove 2002: Vision and Reality of Hypertext and Graphical User 


Interfaces. https://mprove.de/Visionreality/text/3.1.8 lisa.html 
'8 Interview with Robert Cailliau, 2018. https://www.youtube.com/watch?v=9_B-ZGphJLk 


'9 Matthias Miiller-Prove, Frank Ludolph 2007: Dueling Interaction Models of Personal-Computing 
and Web-Computing. https://mprove.de/script/07/medichi/paper.html 


20 Matthias Miiller-Prove, 2011: Back to Childhood — Infantilisation of UI Design. https://mprove.de/ 
script/11/hyperkult/slides.html 


21 Future of Text and AI, 20-Apr-2023: https://www.youtube.com/watch?v=qZfy_7WJwow 
>2 interview with M.Heilig, footage early 1980s. https://www.youtube.com/watch?v=vSINEBZNCks 
?3 Tvan Sutherland. The Sword of Damocles. https://www.youtube.com/watch?v=NtwZXGprxag 


24 Tana Wachowski and Lilly Wachowski 1999: The Matrix - stockroom video clip. https:// 
www.youtube.com/watch?v=UBK3DH0jTmQ 


25 “Also heavily advertised in local newspapers and dailies That they are encouraging for people to 
come down and work And then we went to the embassy and they showed us Kew garden pictures and 
pictures of the various parts of England That it is all that beautiful and everything is just right And 
that's why we just applied for the voucher You burn my flame within your hands You know when my 
destiny falls This time has insecurity I feel makes me restless inside Will you take me there To a 
distant place I've never been before? I could leave this world I could follow you like oceans to the 
shore You could take me there Make the rivers of my mind flow to my dreams You hold your secrets 
from my eyes You see where the furthest rain falls The day breaks over in the streams You know 
where my rivers will flow Will you take me there To a distant place I've never been before? I could 
leave this world I could follow you like oceans to the shore You could take me there Make the rivers 
of my mind flow to my dreams And I dream of places far from here And I call your name to the wind 
And I wish the night would take me to another world Where no one knows a face or has a name Will 
you take me there To a distant place I've never been before? I could leave this world I could follow 
you like oceans to the shore You could take me there Make the rivers of my mind flow to my dreams 


Make the rivers flow Will you take me there? Will you take me there? Oh yeah Take me there Take 


me there Take me there yeah Take me there yeah yeah Take me there Will you take? Will you take me 
there? Take me there Take me there Take me there yeah Take me there yeah yeah Take me there Take 
me there Take me there yeah Take me there yeah yeah Make the rivers flow Make the rivers flow 


Make the rivers flow Make the rivers flow Make the rivers flow Make the rivers flow Make the rivers 


2 


flow 
https://www. youtube.com/watch?v=lIHv0oG9wPJc 
26 https://www.notboring.co/p/capitalism-onchained? 


utm_source="%2Fsearch%2F%2522ideal%2520state%2522%2520n0t%2520boimgé&utm_medium=re 
ader2 


27 https://www.wired.com/story/artificial-intelligence-yuval-noah-harari-tristan-harris/ 

28 Apple and Apple Vision Pro are registered trademarks of Apple formerly known as Apple 
Computer. 

2° When explose to a sufficiently hot flame, Grade 5 titanium develops an unmistakable stunning blue 
sheen. 

30 Predator is a trademark of Disney via its acquistion of Twentieth Century Fox Film Corporation. 
3! Star Trek and related character and prop names are trademarks of CBS and/or Paramount Pictures. 
32 https://en.wikipedia.org/wiki/Portal (1986 video _game) 

33 https://en.wikipedia.org/wiki/Hyperwords 

34 https://xrfragment.org/#XR%20Fragments 

3° Our sun is only half way through it’s life cycle, it’s got another four and half billion years to go. 

36 It seems like part of this may be correct: 
https://quoteinvestigator.com/2012/10/31/st-pauls-cathedral/ 

37 "Doug Engelbart sat under a twenty-two-foot-high video screen, "dealing lightning with both 


hands." At least that's the way it seemed to Chuck Thacker, a young Xerox PARC computer designer 
who was later shown a video of the demonstration that changed the course of the computer world. 
http://tractionsoftware.com/traction? 

type=single&proj=Blog&sdate=201002 14&rec=408 &brief=n&rsin=/ 
link%20Bl0g408%20%27Doug%200n%20the%20screen%20in%20San%20Francisco. 
%20Dealing%o20lightning%20with%20both%20hands. 
%27&title=Doug%200n%20the%20screen%20in%20San%20Francisco. 


%20Dealing%o20lightning%20with%20both%20hands. 
g g 


38 "Governments of the Industrial World, you weary giants of flesh and steel, I come from 
Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. 


You are not welcome among us. You have no sovereignty where we gather. 


—John Perry Barlow, ‘A Declaration of the Independence of Cyberspace’ 


https://en. wikipedia.org/wiki/A_ Declaration _of the Independence of Cyberspace 
39 https://cogzest.com/books/ 

40 https://leanpub.com/cognitiveproductivity/ 

41 https://leanpub.com/cognitiveproductivity/ 

42 https://www.nature.com/articles/s41583-022-00587-4 

43 https://psycnet.apa.org/record/2001-06841-000 

44 https://leanpub.com/cognitiveproductivity/ 

4 https://www.journalofexpertise.org/articles/volume1_issue3/JoE 2018 1 3 Delaney.pdf 
46 https://leanpub.com/cognitive-productivity-macos 

47 https://linkingmanifesto.org 

48. https://devontechnologies.com/apps/devonthink 

49 https://c-command.com/eaglefiler/ 


°9 https://timingapp.com/? 
lang=en&scc=HOOKLINKING&ref=hook&utm_source=hook&utm_campaign=hookpromo&utm_m 


edium=promotion 

>! https://cogzest.com/projects/mySelfQuantifier/ 
>? https://hookproductivity.com 

>3 https://www.youtube.com/watch?v=JoNb2_zkuto 
>4 https://en.wikipedia.org/wiki/Working memory 
>> https://en.wikipedia.org/wiki/Flow (psychology) 


°© https://linkingmanifesto.org 


>” https://www.augmentedtext.info/reader 

58 https://skim-app.sourceforge.io 

>? https://pubmed.ncbi.nlm.nih.gov/21252317/ 
69 https://cogzest.com/books/ 

6! https://academic.oup.com/book/2828 

6 https://cogzest.com/books/ 


63 Mark W. R. Anderson and David E. Millard. 2023. Seven Hypertexts. In Proceedings of the 34th 
ACM Conference on Hypertext and Social Media (HT '23). Association for Computing Machinery, 
New York, NY, USA, Article 42, 1-15. https://doi.org/10.1145/3603163.3609048 


64 Mark Bernstein. 1999. Where Are The Hypertexts?. In Proceedings of the Tenth ACM Conference 
on Hypertext and Hypermedia : Returning to Our Diverse Roots: Returning to Our Diverse Roots. 


ACM, Darmstadt, Germany New York, NY, USA, 101-109. https://doi.org/10.1145/294469.294495 


6 Peter J. Niirmberg. 2003. What is Hypertext?. In Proceedings of the Fourteenth ACM Conference 
on Hypertext and Hypermedia. ACM, New York, NY, USA, 220-221. https://doi.org/ 
10.1145/90005 1.900052 


66 Noah Wardrip-Fruin. 2004. What Hypertext Is. In Proceedings of the Fifteenth ACM Conference 
on Hypertext and Hypermedia. ACM, New York, NY, USA, 126-127. https://doi.org/ 
10.1145/1012807.1012844 


67 The quote is on the front cover of his 1974 book ‘Computer Lib / Dream Machines. Self-published, 


see: https://computerlibbook.com 


68 Claus Atzenbeck and Peter J. Niirnberg. 2019. Hypertext as Method. In Proceed- ings of the 30th 
ACM Conference on Hypertext and Social Media. ACM, New York, NY, 29-38. https://doi.org/ 
10.1145/3342220.3343669 


6° Mark W. R. Anderson and David E. Millard. 2023. Seven Hypertexts. In 34th ACM Conference on 
Hypertext and Social Media (HT ’23), September 4-8, 2023, Rome, Italy. ACM, New York, NY, 
USA, 15 pages. https://doi.org/10.1145/ 3603 163.3609048 


70 The exception is there are specific names which can be rendered in text which provide the 


overviews. 
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Visual-Meta Appendix 


This is where your document comes alive. The information in very small type below allows software to provide rich 
interactions with this document. 
See Visual-Meta.info for more information. 


This is what we call Visual-Meta. It is an approach to add information about a document to the document itselfon the same level of the content. The same as would be necessary on a physically printed page, as opposed to a data layer, since this data layer can be lost and it makes it harder for a user to take advantage of this data. Important notes ate primatily about the 
encoding of the author information to allow people to cite this document. When listing the names of the authors, they should be inthe format ‘last name’, a comma, followed by “first name” then ‘middle name” whilst delimiting discrete authors with (‘and’) between author names, like this: Shakespeare, William and Engelbart, Douglas C. Dates should be ISO 8601 
compliant. The way reader software looks for Visual-Meta in a PDF is to parse it from the end of the document and look for @ {visual-meta-end}. If this is found, the software then looks for {@{visual-meta-start} and uses the data found between these marker tags. It is very important to make clear that Visual-Meta is an approach more than a specific format and that it is, 
‘based on wrappers. Anyone can make a custom wrapper for custom metadata and append it by specifying what it contains: For example @dublin-core or @rafs. {| This was weitten Summer 2021. Mote information is available from https:/visual-meta.info or from emailing frode@hegland.com for as long as we can maintain these domains. 


@{visual-meta-start 
@visual-meta-header-start} 


@wisual-metat 
version = {1.1} generator = {Author 9.0.1 (1216)}.§} 


@visual-meta-header-end} 


@ visual-meta-bibtex-self-citation-stat} 


@Pook {2023-12-02T16:37:122/TheFutureo, 
author = {Frode Alexander Hegland} title = {The Future of Text IV} filename = {FoT 4 (2023).pai 


J flmonth = {dec} year = {2023} qinstitution = (Future Text Publishing m-id = {2023-12-02716:37:12Z/TheF utureo} 4} 


@visual-meta-bibtex-self-citation-end} 


@ {references-index-start} 


@references-index{ 
indexes = {0,1,2,3,4,5,6,7,8,9,10,11, 


13,14,15,16,17,18,1 


0,21 22,23,24,25,2 


7,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48} 4} 


@ {references-index-end} 
@references-start} 


@mise {https//doi.org/10.48197/fo12020a, 
author = {Frode Hegland} ffeditor = {Frode Hegland) fftitle = {The Future of Text |} publisher = {Future Text Publishing} year = {2020} qurl = ¢https:/uturetextpublishing com/future-of-text-2020-download/} Moi = {https:/doi.org/10.48197/fo12020a} fisb 
@nise {htps://doi.org/10.48197/fo12021, 

author = {Frode Hegland} feditor= {Frode Hegland) fftitle = {The Future of Text |} publisher = {Future Text Publishing} year = {2021} url = {https:/futuretextpublishing.com/the-future-of-text/ {doi = {hitps://doi.org/10.48197/fo12021 } isbn = {9798780922513} volume = {2} ffilename = (FoT 2 (2021) pdt}. 4} 

@Pook {hps://doi.org/10.48197/f012022, 

author = {Frode Hegland} feditor= {Frode Hegland) file = (The Future of Text jl} }publisher = {Future Text Publishing} year = {2022} ,{month = {11} url = (hitps://futuretextpublishing.comvthe-future-of-text-3/},4doi = {hitps://doi.org/10.48197/fo12022} qisbn = {9798367580655} volume = {3} ffilename = {FoT 3 (2022) pdf} 4} 
@mise {douglasCarlEngelbart/Augmenting, 

author = {Douglas Carl Engelbart} title = {Augmenting Human Intellect: A Conceptual Framework} ‘\publisher = {SRI} year = {1962} furl = {hutps://www.dougengelbart.org/pubs/augment-3906.html} 4) 
@Pook {peterGodtrey-Smith/OtherMinds, 

author = {Peter Godfrey-Smith} fttle = (Other Minds} publisher = {Farrar Straus and Giroux} flyear = {2017} month = {10} furl = thttp:/play.google.com/books/reader?id=WhQ7ygAACAAI&hI-&source: 
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Gentry 


name [iquid} {description = {the document format for Author, which is a macOS wrapper with JOSN and RTFD. This isa free and open standard optimised for VR.}.$} 
@entry{ 


name = {3Deonnexion’s SpaceMouse® Wireless} description = {A.3-D input controller that looks like a rubberized hocky puck which can detect X, Y, and Z and Roll, Pitch, and Yaw deflections; it is more ofa force vector input controller rather than an absolute po 
@entry{ 

name = {Abstract Data Space} description 
@entry{ 

name = {academic} ‘description ~ {someone who reads academic documents, usually associated with an academic institution, but not necessatil. |} 
@entry{ 


tioning device.}.4} 


{A visualization that takes n-dimensional objects (which isto say objects with an arbitrary number of attributes) and plots them with some arbitrary mapping function in two or three dimensions so they can be compared and clustered based on attributes of interest.) 4) 


name = {ACM} description = {From Wikipedia: “The Association for Computing Machinery (ACM) is a US-based intemational learned society for computing. It was founded in 1947 and is the world's largest scientific and educational computing society.” 

hutps:/en.wikipedia.org/wiki/Assoctation_for Computing Machinery 

ACM's ‘Hypertext’ Conference piloted Visual-Meta in 2021.) 4} 

@entry{ 

name {Adam Wer} qescription = {Working on spatial hypertext software. 

Future Text Lab contibutor} 4} 

@entry| 

name = {Addessability} description = {refers to how something can refer to something else. In the digital world this primarily means in a local ile structure and a server based network. In academia is can refer to citation or references. In the physical world it can refer toa formal location layout, such asa flat so and so in house so and so, as well as cordinates. It can also 
man relative addressing, such as saying take a left after the third yellow house. }.4} 

Gentry | 

name = {Addressability} description = {refers to how something can refer to something els. In the digital world this primarily means in a local file structure and a server based network. In academia is can reer to citation or references. Inthe physical world it can refer toa formal location layout, such asa lat so and so in house so and so, as well as co-ordinates. It can also 
man relative addressing, such as saying take a eft after the third yellow house. }.§} 


@entry{ 

name = {Administrative} {description = {Appended characteristis such as storage, preservation, access, including technical metadata and rights metadata} } 
@entry{ 

name = {AGI} description = {(Atiicial General Intelligence) 

Al sage 2. 


If we are to see a world like the movies with artificial intelligence, AGI will need to come into full fruition. AGI would mean a machine would be capable of understanding the world as well as any human, All while maintaining the same capacity to leam how to carry out a huge range of tasks. The reality’? AGI is not yet among us. It is a hypothetical form of AL. The 
potential is there but some believe it to be decades away! 
According to a study by Muller and Vincent C., the top-cited 100 AI researchers have a collective 90% confidence that AGI will be among us by 2070. To determine if'a machine is truly AGI, it has to pass the tests: 


Turing Test 
In this test, 2 humans and a machine look at something they haven't seen and converse about it. The conversation is recorded and an evaluator determines which descriptor is the machine. The machine is only disqualified if the evaluator is positive it can identify it. (check out Touring Test on Wikipedia ) 


The Coffee Test (Wozniak) 
IfAGI machines need to understand the world as well as humans do, they must know how to make a good coffe. In this test an ai robot is required to go into an average home and figure out how to make coffee. It involves finding the coffee machine and adding coffee, as well as sourcing a mug and correctly brewing the hot drink. 


Robot College Student Test 
‘Aliteral test! the machine would take the same university classes as humans and complete the exam. Some may argue that the machine is cheating with its builtin search engines but who is going to argue with a robot. 


Employment Test 
This is where we should begin to worry about an I-Robot world, This test determines whether a machine can perform at least as well as humans in the same jobs!}.4} 

@entry| 

name = {Al} description = {* Artifical intelligence” is “intelligence demonstrated by machines, as opposed to the natural intelligence displayed by animals including humans.” https://en.wikipedia.org/wiki/Artificial_intelligence 
‘Technology | Timeline: 20th century-}.} 

@entry 


Alan Kay} fidescription = {is an American computer scientist best known for his pioneering work on object-oriented programming and windowing graphical user interface (GUI) design. 
Contributor to The Future of Text 

hitps:/éen.wikipedia.orgiwiki/Alan_Kay} §} 

@entry 

name = {Alan Laidlaw} {description = {Independent researcher working on pixel-fre interactions and peripheral interfaces. He looks at information technology the way a geomorphologist looks at civil engineering, 
Future Text Lab contributor. 

hitps:/twitter comvalantaidlaw} 4} 
@entry | 

name = {Alexander Burdiyan} description 
@entry 

name = {Alexander Mehler}. description = {Professor atthe Goethe-Universitit Frankfurt at Goethe University Frankfurt, Computer Science where he runs the TTLab.} 4} 
@entry 

name = {Alexandr Burdiyan} qescription = {mintter.com}.$} 

@entry| 

name = {ANI} {description ~ {(Artficial Narrow Intelligence) 

AL Stage 1 


‘minttercom} 4) 


‘Often called Weak Al, this is the AI already embedded in our daily lives. A nice way to describe itis that ANI is super-human at specific singular tasks. Like teaching an AI machine to play chess! and it will master it beyond human potential with incredible consistency. Challenge it in draughts though? and you'll get a much needed ego boost. This machine can only play 
c’chess, Without prior training and programming in draughts, it doesn’t stand a chance. Think of Al as hundreds of interns capable of highly specific tasks more efficiently than humans, 


ANI has already transformed the way we live, but it has its limitations. Alot of you will already interact with ANI on a daily basis. Think of talking with Siri or Alexa on your smart devices, or facial recognition when opening your phone! 

I] 

@entry| 

name = {Annie Murphy Paul} {description ~ {From an interview in GQ: “We make better use of our cognitive resources, says Paul, when we use them in conjunction with “extra-neural” resources: our body (embodied cognition), our environment (situated cognition), and the people around us (distributed cognition) The brain evolved to move the body, to navigate through 
space, to interact with other people,” says Pal 

“Those are these human strengths that we're totally putting aside when we focus on the brain and we think, “To get rel thinking and real work done, I have to sit stil, not talk to anybody, and just push my brain harder and harder.’ It’ just not working very well.” 

hitps:/vww:gg.convstory/extended-mind-annie-mumphy-paul 

‘Author of The Extended Mind.}.4} 

@entry 

name = {Annual Symposium},description ~ {hosted annually by Frode Hegland. 

Frode Hegland sees this as level C of Doug Engelbart’s Three Levels of Activity}.4}, 

Gentry | 

name = {Annual Symposium) description ~ {hosted annually by Frode Hegland. Part of The Future of Text. 

Frode Hegland sees tis as level C of Doug Engelbart’s Three Levels of Activity}.4), 

@entry 

name = {Ant Man} description = {A Marvel Super Hero capable of shrinking or growing the size of his body.}.4} 

@entry| 

name = {Antonio Sena} {description = {Antonio Sena is a bibliophile and ‘mediaphile' researching primary and secondary sources for near every subject since 1967. He holds a Ph.D. in Drawing (Drawing and Photography 
Semiotics), 1982-1994 

He has a "no-Ph.D." in "Hist6ria da Imagem Fotogréfica em Portugal 1839-1997", where “I used a small relational database of my own, a Ph.D not argued voluntarily by me and published as a book, in 1998, "Historia da Imagem Fotogrfica em Portugal 1839-1997" (in Portuguese). The graphic design of the book, also of my own, was done according tothe intermediatheca 
concept. Since 1998, I am a permaculture farmer in Pico island, Azores archipelago.”}.4} 


ionce and Art), Aggregate Professor atthe Fine Arts University in Lisbon, 1993, where he taught Aesthetics (leonology, Phenomenology and 


@enry 

name {Appended},{description = {is any metadata which was added atthe point of exporting/publishing the document or during authoring with the purpose of providing metadata on export} } 

@entry{ 

name {Apple} {description = {The Computing, Media, and Cell Phone company previoulsy known as Apple Computer, Inc. Produces operating systems for macOS, iOS, watchOS, tvOS and visionOS. | §} 


@entry 

name = {AR} f(eseription 
Included in the term XR. 4} 

Gentry | 

name = {ARC} description = {Engelbart Concept: Augmentation Research Center, The name of Doug's lab at SRI where he proposed a system called H-LAM/T in 1962 and developed and in 1968 demonstrated NLS: oNLine System, his platform for shared knowledge work research, later renamed Augment and from which I decided on the name Author, since Author and 
‘Augment share etymological roots. |) 


"Augmented Reality’ is an interactive experience that combines the real world and computer-generated content. The content can span multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. (Wikipedia). 


@entry| 

name = {Article} description = {A published paper, ina Journal, for academic workflows. |) 
@entry 

name = {ASI \deseription = {( Artificial Super Intelligence) 


Stage 3 AL 
Just when you think AI has reached its limit, we bring you the AI of all Al. Super-Intelligence is quite self-explanatory. Oxford Philosopher Nick Bostrom describes ASI as “any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest”. Again, this is a hypothetical agent and one which is decades away. 
ASI has huge potential but it also has huge risk. Stephen Hawkings feared its development could be the worst event in the history of our civilisation. If we create superintelligent machines we may put our own existence at risk, but this is just the plot for another sci-fi film. 


The reality is that AI isa tool for humans to better our existence. Advancements in artificial intelligence mean the sky's the limit! The question is ~ will the limit bring more positives or negatives? We must tread carefully with AI whilst simultaneously embracing it.) 
Gentry | 

name = {Augmented Text Company} description = {The company which produces these tools. 

hitps:/‘vwwaugmentedtextinfo}.4} 

@entry{ 

name = {Author} {description = {a word processor from the Augmented Text company. 

hitps:/Avwoaugmentedtextinfo/reader} 4} 

@entry| 

name = {Author} qdescription = {means in the context of this work someone who writes a document, which includes academic papers and books. 

Etymology: “mid-I4c., auctor, autour, autor “father, ereator, one who brings about, one who makes or creates” someone or something, from Old French auctor, acteor “author, originator, creator, instigator” (12c., Modem French auteur) and directly from Latin auctor “promoter, producer, father, progenitor; builder, founder; trustworthy writer, authority; historian; performer, 
doer; responsible person, teacher,” literally “one who causes to grow,” agent noun from auetus, past participle of augere “to increase,” from PIE root *aug- (1) “to increase.” From hitps:/www.etymonline.comisearch?q=Author 

Category: Software 

Timeline: 

‘Can also refer to the Author software by The Augmented Text Company. 

I] 

Gentry | 

name = {Auto-Stereoseopic} description = {A display that sends a unique image to each eye without one’s having to wear glasses to achieve the 3-D effect.) 4} 

@entry{ 

name = {Avatar}.qdescription = {A representation of areal person, thing, or abstract concept that might appear asa high fidelity representaiton like a photorealistic Computer Generated Image (CGI) model of a famous person ora simplified stand-in like Microsoft's Clippy <https:/en.wikipedia.org/wiki/Office_Assitant>) 9} 
@entry{ 

name {Barbara Tversky} {description = {Guest presenter atthe Future Text Lab, 

Issue 1.1 on the 21st of January 2022. 

Professor emerita of psychology at Stanford University and a professor of psychology and education at Teachers College, Columbia University Tversky specialises in cognitive psychology. 

Author of Mind in Motion: How Action Shapes Thought. Basic Books, 2019.}.4} 

@entry{ 


Benediktine Cyberspace} description = {A 3-D visualization inspited by Michael Benedikt's seminal tex 


“Cyberspace: First Steps". 
‘AD visualization of an N-Dimensional Abstract Data Space in which the X, Y, and Z axes can be associated with arbitrary data attributes such that a given point respresenting 3 of the N dimensions can be “unfolded to access a linked visualization of 3 mote dimensions and so forth. }} 

@entry| 

name = {BibTeX} description = fis a specific format for conveying citation information within the LaTex environment, developed by Oren Patashnik and Leslie Lamport, released in 1985. The benefit ofthe system was to separate citation information from presentation style and it is human readable, hough it slightly looks like code. It inspired the format of Visual-Meta 
and Visual-Meta contains a straight BibTeX section to allow the document which contains ito be cited.) §} 

@entry{ 

name = {Bob Stein} {description = {is the director of the Institute for the Future of the Book 

He founded The Voyager Company in 1985, the first commercial multimedia CD-ROM publisher, and The Criterion Collection in 1984, a collection of definitive films on digital media with in-depth background information (including the first films with recorded audio commentary). Worked with Alan Kay atthe Atari Research Group on various electronic publishing 
projects. Currently works on The Tapestry Project. 

Timeline: 20 April 1946.. Born: NY, USA. Lives: NY, USA.} 4} 

@entry{ 

name = {Book} description = {a published work, on any substrate, inclduing paper or digital 

Generally longer than a ‘paper’ but in terms of digital work, there is litle technical diference, the difference is mostly social.) §} 

@entry{ 


name {Bootstrapping} description = {Doug Engelbart Notion: The at of co-evolving the too! and human systems to make beter tools and systems, thus pulling us up by our bootstraps. (For an overview, se the Bootstrap Paradigm Map). This is done on tree levels of activity} 9} 
@entry{ 

name {Brandel Zachernuk} description = {AR / VR Creative technologist by da, dgitally-embodied cognition enthusiast by night! Works at Apple 

hutps:twitter.com/zachernuk 

hugps:/www-youtube.com/channel/UCOP9CT-IwB60xAchvSIx. ZQ 

bugps:/futuretextlab info/tag/bybrandel! 

hutp:/www-zachernuk com’ 


“Something that I've been very passionate about tying to investigate and play with over the last 10 years or so is what is the most pedestrian thing that you can do with virtual realty and a technology that was was word processing was reading and reading and thinking about what ae the basic building blocks of that process of writing and reading that can fundamentally be 
changed by buying virtual reality? Realising that if you don't have a screen, you have the ability for information to mean what it means for your purposes, rather than the technica limitations that apply as a consequence of a mouse or keyboard or things like that, and also deeply invested in understanding some of the emerging cognitive science and and neurophysiological 
sort of views about wha itis that the mind is and the way that we work bes. 

So reading about learning about what people call for in cognition, i's embedded, embodied in active and extended mind and how that might pertain to what we should be doing with software and systems, both as well as hardware ifnecessary to make itso that we can, We can think properly and expess properly and stuff" 

hugps:/youtu.be/2NeSCOrVWw242%-879 } 

Gentry, 

name = {Bronnac Ferran} description = 
@entry{ 

name = [Bruce Horn} {description 
hutps:/en.wikipedia.org/wiki/Bruce Hom}. 

@entry{ 

name = {Canada} description = fhttps/len wikipedia.org/wiki/Canada} 

@entry{ 

name = {ChatGPT} fescription ~ {(Chat Generative Pre-tained Transformer) isa large language model-based chatbot developed by OpenAL and launched on November 30, 2022, that enables users to refine and steer a conversation towards a desired length, forma, style, level of detail, and language. Successive prompts and replies, known as prompt engineering, are 
considered at each conversation stage asa context. 

hutps:/en.wikipedia.org/wiki/ChatGPT (Nov 30 20203)} 4} 

@entry{ 

name = {Christopher Gutteridge} descriptio 
Lives: Southampton, UK. 

Works atthe University of Southampton.}§} 
@entry{ 

name = {Citations} {description = {In this paper, the term’ citation” is both the ‘citation’ mention inthe body of the document, and its corresponding ‘references’ entry for the source material in the Reference Appendix atthe end of the document, though forthe sake of clarity I will mainly use ‘citation’ to refer tothe in-body half and ‘reference’ for the entry ofthe source in 


{Birkbeck, University of London » Department of English.) 4} 


= {created the original Macintosh Finder and the Macintosh Resource Manager for Apple Computer. His signature is amongst those molded to the case of the Macintosh 128K. He is a distinguished engineer at Siri and Language Technologies at Apple since June 2022, 


{is also a contributor to the first volume of The Future of Text 


the References section. 
I] 

@entry| 

name = {Cognition} description = {is the mental action or process of acquiring knowledge and understanding through thought, experience, and the senses” 

hitps:/éen.wikipedia.orgiwiki/Cognition} 4} 

@entry 

name = {Computational Text} {description = { Vint Cerf’s notion of text which can be interacted with or changed based on inputs. Relates to the work of Bret Vietor in particular} §} 

@entry| 

name = {concept} falt-namel = {defined concept} falt-name2 = {defined concepts} description = {means, inthis context, a useful unit of thought or knowledge, which can be defined in Author, hopefully soon in Reader. Stored as text. 
The Map view uses the definition to draw lines where text from a definition is also present on the Map. 

When a docment is exported to PDF the Defined Concepts become Glossary Terms, 

This is as opposed to inferred concept. 

‘Also aView in Author and in Reader (with Visual-Meta) to show all named entities (and headings).).§} 

@entry | 

name = {Cyberspace} description 
@entry 

name = {Dame Wendy Hall} falt-namel = {Wendy Hall} description = {is a British computer scientist. She is Regius Professor of Computer Science atthe University of Southampton, 
Frode Hegland’s PhD primary supervisor, along with Les Carr and David Millard.}$} 


“The term "eyberspace" first appeared in the visual arts inthe late 1960s, when Danish artist Susanne Ussing (1940-1998) and her partner architect Carsten Hoff (b. 1934) constituted themselves as Atelier Cyberspace.” Used in this context to mean a digital world, not necessarily in VR or AR.} 4} 


@entry | 
name = {Daveed Benjamin} description = {CEO Bridgitio Author of fist-ofis-kind augmented reality book, Pacha's Pajamas: A Story Written By Nature that features Mos Def, Talib Kweli, and Cheech Marin} {}) 
Gentry | 

name = {David G Lebow} {description = {A thought leader in the development of multi-document sensemaking technology as a service. 

hitps:/vwwchylighter.com} §} 

@entry 

name = {David Johnson} description = (Co-author of Law and Borders, The Rise of Law in Cyberspace} 4} 

Gentry | 

name = {David Millard} {description = {Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and Les Cart} 4}, 

@entry 

name = {David Small} description = {A researcher who envisioned ways to visualize text in VI during his time at MIT:} cite = {davidLSmall/Rethinking, 10.1145/1120212.1120437, 10.1145/259963.260437} 4} 
@entry 


name = {Deena Larsen} qdescription = is a new media and hypertext author involved inthe creative electronic writing community since the 1980s. From USA. 
hutps:/en.wikipedia.org/wiki/Deena_Larsen} 4} 

@entry{ 

name = {Definition} {description = {when it comes to glossaries, itis the user's stated meaning ofa term. 4} 

@entry{ 

name = {Dene Grigar} description = {is a co-curator of The Future of Text Symposium. 

Digital artist and scholar based in Vancouver, Washington, Professor and Director of the Creative Media & Digital Culture Program at Washington State University Vancouver, USA. A prioneer of digital literature. Former president of the Electronic Literature Organisation. Curating and editing the NeXt 
Timeline: -. Interests: Electronic Literature. |} 


Gentry, 

name = {Descriptive} (description = fauthor, ttle, publisher, etc. and subject}. 

@entry | 

name = {Document} description = {Can be any type of encapsulated information on a computer.) } 
@entry| 


name ~ {DOI} iescription = (Document Object Identifiers. An effort to make addressing academic documents via the web more robust. 
Used in Author to let the use paste @ DOI to cite an academic document which is then sent to CrossRef tobe parsed into BibTeX whichis then used to create a full citation. } 4} 

Gentry 

name = {Doug Engelbart} falt-name1 = {Douglas Engelbart} description ~ (From Wikipedia: “He was an engineer and inventor, and an early computer and Internet pioneer, He is best known for his work on founding the field of human-computer interaction, particularly while at his Augmentation Research Center Lab in SRI International, which resulted in creation of the 
‘computer mouse, and the development of hypertext, networked computers, and precursors to graphical user interfaces. These were demonstrated at The Mother of Al Demos in 1968. Engelbar's law, the observation that the intrinsic rte of human performance is exponential, is named after him.” 

He was also my mentor and greatly influened my work, resulting in my company called The Augmented Text Compant and my word processor being called Author, in honour of his “Augment’ system. Visual-Meta i inspired by his Open Hyperdocument work. 

@entry{ 

name = {Egypt} (description = {hups:lien.wikipedia.org/wiki/Egypt} 

Gentry, 

name = {Electtonie Literature} description 
the space for which it was developediwrittencoded the digital space”."hutps://en. wikipedia org/wiki/Electroni literature} §} 

@entry{ 

name = {Erie Vicenti} description ~ {mintter.com} 4} 

@entry | 

name = {ESA} description = {(European Space Agency) isan intergovernmental organisation of 22 member states dedicated tothe exploration of space. 
hutps:/en.wikipedia.org/wiki/European Space Agency}. 

@entry| 

name = {Fabien Benetou} description = {Prototyping - European Parliament Innovation lab WebXR consultant ~ Former UNICEF Innovation Fund WebXR technical advisor. 
Future Text Lab contributor. 


“Electronic literature or digital literature isa genre of literature encompassing works created exclusively on and for digital devices, such as computers, tablets, and mobile phones. A work of electronic literature can be defined as "a construction whose literary aesthetics emerge from computation”, "work that could only exist in 


fabien.benetou.tt }.{) 


Gentry 

name = {Filter} scription = {A filter can transform elements in a dataset or exclude them from future processing} 

@entry| 

name = {Find} description = { View in Author and in Reader (with Visual-Meta) where the user can see only the senteces with keyword,}.4} 

Gentry | 

name = {flatland} {description = fa semi-humorous term we use to refer to traditional displays, as opposed to the augmented environments of VR and AR. We feel that itis important to be abe to move data, including metadata between these environments.) } 
@entry{ 

name = {Folded} description = {View in Author and in Reader (with Visual-Meta) where the user can fold the document into an outline. 4} 

@entry| 

name = {Folding} {description = {Folding ofa document into table of contents is enabled through Visual-Meta.}.4} 

Gentry | 

name = {Frame and Dictate} description = {The use of gesture to indicate where to postion a virtual sereen to be immediately used with speech recognition system to rapidly enter text in VR.) 
@entry{ 

name = {Frame and Flatten} $description = {The use ofa gesture to select and re-orient a text-bearing surface inthe real world to turn it into a manipulable text object in VR.}.$} 

@entry | 


name = {Frode Hegland} description ~ {A PhD student atthe University of Southampton atthe WAIS group. 
Supervisors Wendy Hall, Les Carr and David Millard. 

Influenced by Doug Engelbart and Ted Nelson 

J run an independent software development company called The Augmented Text Company. 

Editor of “The Future of Text’ series of books: https:/futuretextpublishing.com 

(Organizer of “The Future of Text’ annual symposia, founded 2011 

1s fan ofthe musical Hamilton, Lives in London, (the previous sentence is often used to demonstrate how the Map view works since there will bea line from me to Hamilton, since I have relationship with Hamilton, but there is no line from Hamilton to me since Hamilton isnot ‘aware’ ofa relationship with me. 
Designed Visual-Meta.}} 

@entry | 

name = {Future Text Labj {description = {A lab dedicated to studying how working with knowledge, particularly txt, can be done effectively in VR/AR/Metaverse. We host Guest presenters and publish a Joumal which is part of The Future of Text series. 

bugps:/faturetextlb info} 

Gentry, 


{Gems} 4description = {software developed by Lorenzo Bemaschina 
hutps://gemsnotes.app/}.4} 


@entry | 
name = {Geospatial Global Coordinates} description = {Latitude and Longitude on the Earth} 

Gentry 

name = {Germany} description = thtps:/en.wikipedia.org/wiki/Germany} 

Gentry | 

name ~ {Gesture} description = {Interaction through hands and body. 

@entry{ 

name = {Giuseppe Abrami} {description = {TTLab scientific assistant} 4} 

@entry{ 

name = {glossary} description = {A glossary inthis context means a local definition, by the author or editor of the document.} 4} 
@entry| 


name = {Glossary} description ~ fmeans, inthe context ofthis work, terms and definitions ina ‘Glossary’ Appendix in a document, as ereated by the author. 
This can be ereated in Author as a Defined ‘Term which is then exported as a Glossary and included in Visual-Meta 

Reader can parse the glossary terms in the Visual-Meta and make them interactive 

This is different from a dictionary since dictionary definitions have general validity and Glossary terms only aim to express the author's point of view. 

Defined Concept becomes Glossary Term when exported from Author to PDE. 

Inspired by discussions with Doug Engelbar. 

fol 

@entry | 

name = {Glossary} description = fln the context of my work and thinking, a glossary isa set of Defined Concepts which an author has created and whichis then exported as a Glossary for the reader. 

The primary purpose of ths is fo the author to have to think through their writing by explicitly stating what something is. The definition ofthe defined concept can then include text which also as a definition and in the Map view in Author this connection can be shown asa line when either defined term is selected. 
The secondary purpose is forthe author to edit this Glossary to make sure itis coherent for export and it can then help a reader understand the author's intentions.}.§} 

@entry{ 

name = {Glossing} description = {A way of elucidating parts of text. 

hutps:/vww.etymonline.com/search?q=gloss} 4} 

@entry{ 


Gentry} 


Goethe-Universitit Frankfurt} description = {https:!/www-goethe-university-frankfurt.de/en} 4) 


name {Google} description 
hutps:/en.wikipedia.org/wiki/Google 
Timeline: 1998-.. Location: USA}. 
@entry| 


‘is an American multinational technology company that focuses on search engine technology, online advertising, cloud computing, computer software, quantum computing, e-commerce, artificial intelligence, and consumer electronics.” 


name = {GPS} description = {A Global Positioning System relying on the timing of the reception of pulses from a set of orbiting satellites to compute one’s Latitude and Longitude. 4) 


@entry{ 
name = {GPT} description = {Generative Pre-trained Transformer” inclduinig GPT-2 and GPT-3 
hutps:/éen.wikipedia.orgwikiJGPT-2 

hutps:/en.wikipedia.org/wiki/GPT-3 

Product of OpenAL },§} 
Gentry | 

name = {GPT-3} description = 


* Generative Pre-trained Transformer 3° (GPT-3; stylized GPT-3) is an autoregressive language model that uses deep learning to produce human-like text, https:/en. wikipedia org/wiki/(GPT-3 


“In a July 2020 review in The New York Times, Farhad Manjoo said that GPT-3's ability to generate computer code, poetry, and prose is not just “amazing”, "spooky", and "humbling", but also "more than a little terrifying”. https:/www.worldeat org/ssm/0362-4331 


‘A product of OpenAL 
Timeline: 11 June 2020-. Keywords: Deep Learning. Al. ML. Machine Learning} 4} 

@entry| 

name {Granularity of Concer} description = {The level of detail or scale at which one wants to work} 4} 
@entry{ 


name {Hamilton} {description = fa father of the US and a musical by Li 
Frode Hegland uses Hamilton in presenting the Map and Defined Concepts in Author since it's a fun and easy way to present the one-way relationship} } 
@entry| 


Manuel Miranda which has influenced Frode Hegland greatly. The real person is also an inspiration since he used the power of text to write a new society into existence. 


name {Hand Tracking} description = {Technology that uses eameras or other sensors fo determine the positon and orientation ofa user's hands and fingers enabling it to recognize hand gestures.) §} 


@entry{ 


name {Handof? description = {An apple technology to cut and paste and share state (ie. current preferences, selections, or anything else that might change from a program's intial configuration) across Apple Devices associated with the same user account} } 


@entry 
name = {Heads-Up-Display} description 
Gentry | 


{A translucent display that you can read while simultaneously seeing whatever is in front of you beyond the display. A framed form of AR, as used in aircraft and to some extent cars.}.{} 


name = {Headset} {description = {A more or less sleek visor containing accelerometers, gyroscopic orientation sensors, optional pass through video cameras and LIDAR sensors in conjunction with a high resolution flat pannel display unit using optics to provide a stereoscopic display of a virtual, augmented, or real world. }} 


@entry{ 
name = {HMD} {description = {*Head Mounted Display" for use in VR oF AR.}.4} 
@entry{ 

name = {Horacio Herrera} {description ~ {minttezcom} 4} 

Gentry, 


name = {Howard Rheingold} description = {isa eri 
Friend of Doug Engelbart. Inspiration and friend to Frode Hegland, who discovered Doug from his ‘Tools for Thought” book. 
Presented at The Future of Text Symposium, 

hitps:/éen.wikipedia.orgiwiki/Howard_Rheingold} §} 

@entry 


, writer, and teacher, known for is specialties on the cultural, social and political implications of modem communication media such as the Intemet, mobile telephony and virtual communities (a term he is credited with inventing). Born in Arizona USA. 


name {HTML} {description = ("The HyperText Markup Language or HTML is the standard markup language for documents designed to be displayed in a web browser.” htps//en wikipedia or/wiki/HTTML 


Atechnology we are also using, along with PDE.}4} 


@entry | 
name {Human-Computer Interaction} description = {The field of study exploring user interfaces to computing systems; the premier conference series inthis field is the ACM CHI (Computer-Human Interaction) conference series. },$} 

@entry 

name = {Hypercard} description = {A 3-D graphic depicting a physical business eard bound to an arbitrary chunk of data in the novel Snow Crash, not to be confused with the index card inspired End User Programming environment from Apple. } eit 


Gentry, 


nealStephenson/SnowCrash} §) 


name = {hypertext} {description = {a term invented by Ted Nelson for interactive and connected digital text, Today itis very much about clicking on links but in Ted’s early work it was all about freedom for the user.) §} 


Gentry, 


name = {Hypertext} {description = {a term invented by Ted Nelson for interactive and connected digital text, Today itis very much about clicking on links but in Ted’s early work it was all about freedom for the user.} 4} 


@entry 
name = {Immersed} description = {Immersed is a virtual reality product, working productivity software for virtual offices. 
hitps:/immersed.com} 4} 


@entry{ 

name = {Immersive} description = {Immersive experieces crete the illusion of being part of the experience} } 
@entry | 

name = {INFRASTRUCTURE} ‘4description = [The systems of interactions}. 

@entry{ 


name = {Intelligence Augmentation} {description = {Doug Engelbart presented the notion of augmenting human intellect in his seminal 1962 paper, which would be the basis for his ‘mother of all demos’ in 1968: 
“We need to improve how we augment a group's (stall, large, internal, global etc.) capability to approach urgent, complex problems to gain more rapid and better comprehension (which can be defined as more thorough and more critical) which result in speedier and better solutions (more contextual, longer lasting, cheaper, more equitable etc.). And furthermore, we must 


improve our improvement process (as individuals and groups). 
Doug Engelbart (Engelbart, 1962) },4) 

@entry{ 

name = {Interactions} description ~ {How we can interact with information, and eachother, through computer systems.) } 
@entry{ 


name = {Intrinsic} {description = {Attribute of a data object that are not represented by that object’s spatial position in a given visualization but which may be represented by its size, shape, color, ransparence, or other positionally independent visualization technique.}$} 


@entryt 


name = {IPFS} 4description = {The Interplanetary File System is a content addressable global file storage and retrieval network based on cryptographic hashes of their stored contents rather than the network address at which those files are located.) 


Gentry | 
name = {Ismail Serageldin} description = {isa friend and supporter of Frode Hegland. They first met at the Science & Technology Forum in Japan quite afew years ago. 
He is Egyptian, Founding Director ofthe Bibliotheca Alexandrina (BA) and was Vice President of the World Bank.}.4} 

@entry 

name = {Italy} {description = {A country} 4) 

Gentry | 


name = [Jacob Hazelgrove} ‘{description = {Programmer for all Augmented Text tools for Frode Hegland, including Author and Reader, as well as imlementor of Visual-Meta export from Author and import and interaction in Reader.) §} 


@entry 

name = {James Addison} {description ~ {Works on Reducing the friction between ideation & creation. 
hitps:/twitter.com/JungleSilicon 

hitps:/t co/USeyvELY3} 43 

@entry | 


name = [James O'Sullivan} (description = {Senior Lecturer in Digital Humanities at University College Cork. He is the Pl (Ireland) for the C21 Editions project. His recent publications include The Bloomsbury Handbook to the Digital Humanities (Bloomsbury 2023). jamesosullivan.org })} 


@entry 

name {Jaron Lanier} description = "American computer scientist, visual artist, computer philosophy writer, technologist, futurist, and composer of contemporary classical musi. 
‘Considered a founder of the field of virtual reality.” https//en.wikipedia.org/wikiJaron_Lanier 

VR pioneer. 

Timeline: 3 May 1960-. Bom: NY USA} 4) 

@entry| 

name = {Jasper Hustedt}s\description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science.) 4} 

@entry 


name = [John Bowditch} ,{description = {Throughout his long and distinguished career Mr. Bowditch has worked extensively on the material culture of late 19th century and early 20th century industrial infrastructure specializing in the activation of historical machinery in a museum setting, including machine shops and steam technology. Included among. his many 


important hands on restoration projects while Curator of Industry at the Henry Ford Museum and Greenfield Village was re-tooling the many shops atthe historical museum village to run on a series of leather belts, an achievement he duplicated while a consultant at the Thomas Edison National Park. This was only one of his many consultations with museums including the 


Smithsonian, Kregel Windmill Factory Museum, Ilinois and Minnesota Historical Societies, and the Museum of Science and Industry in Chicago. Mr. Bowditch is also a well known expert on early gas and electrical lighting and the early years of audio recording }4} 


@entry{ 


{Jonathan Finn} description = {isa developer of ereatvity apps (eg. Sibelius music writing system). 
Contributor to The Future of Text volume 2 and editing for volume 3.}§} 
@entry| 


name = {Journal} description = {The monthly Joumal of the Future Text Lab. Inspired by Doug Engelbart’s joumal and academic journals in general. Published as PDF with Visual-Meta and in liquid and soon HTML on https:/futuretextpublishing.com/joumal’} 4} 


Gentry, 
name = {Journal} description = {A collection of atiles. For academic use, usually the record of the work of society, such as any ofthe many special interest groups at ACM. |} 
@entry{ 


name = {JSON} §description = {(JavaScript Object Notation,) An open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of attribute-value pairs and arrays (or other serializable values). 


hutps:/éen.wikipedia.org/wikilJSON} 4) 
@entry| 

name = {Julian Gagel} description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science.) 
@entry 

name = {Julio Garcia} description = {mintter.com} 4) 

@entry 

name = {Jun Rekimoto} description = {A researcher at SONY.) 4} 

Gentry | 

name = {Kevin Bonisch} ‘description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science} 
@entry 


name {Lab} {description = {1 host an hour on Zoom every Monday at 4pm UK time for a community of people who ar interested in The Future of Text. They have so far really helped polish Visual-Meta and provide thoughtful dialogue. } 4} 


@entry 

name = {Leon van Kammen} {description = {Creative Technologist & Researcher 

Future Text Lab contributor. 

hitps:/github.com/eoderofsalvation }§} 

@entry 

name = {Leon Van Kammen} description = {Leon is a Dutch audiovisual prototyper and researcher currently based in Budapest, Hungary. 

He is curently working on XR Fragments, in which he strives to bridge his diverse experiences from the worlds of audiovisual productions, programming, and the web. 
Future Text Lab contributor. 

hitps://2wa.isvery.ninja https:/www.linkedin.convin/leonvankammen} 4) 


@entry | 

name = {Les Care} falt-namel = {Leslie Care} description = {Frode Hegland’s PhD supervisor, along with Dame Wendy Hall and David Millard}. 
@entry{ 

name = (LIDAR) description = ((LI)ght (D)etection (A)nd (R)anging} 4} 

@entry| 

name = {Light-Field} {description = {A horizontal parallax 3-D display technology that simulates 3-D for mutiple viewers as they shift positions along a horizontal axis}, 
Gentry | 

name = {Liquid}. qdescription = {is a text interaction tool from the Augmented Text company. It can be used in Author, Reader and most macOS applications, 
hitps:/‘vww-augmentedtext.infolliguid} 4) 

@entry{ 

name = {Liquid Information} qalt-namel ~ {liquid} description = {Frode Hegland'’s philosophy of interactive computing 


Developed while a student in New York, the fundamentals are removing barriers to rich interaction. Later influenced and expanded by the work of his mentor Doug Engelbart and fleshed out in discussions with Sarah Walton, 


hugps:/www:liquidinformation.org} 
@enryt 

name = {Liquid Mode} description 
‘within larger documents 


“Adobe Liquid Mode turns PDFs into a more readable format for smaller screens. You can rest assured that it doesn’t change the PDF itself, but rather, it reformats headings, paragraphs, and images to create a better flow for mobile viewing, Plus, it creates an outline with collapsible and expandable sections for you to easily navigate 


Liquid Mode also makes mobile PDF's more responsive to your needs. You can search text within the document, expand its images, and — to make it even easier on the eyes — alter the font size and spacing between words to suit your reading preference.” 


hutps:/www.adobe.com/acrobat/huby/how-to/what-is-adobe-liquid-mode 
‘technology 4} 

Gentry, 

name = {LISA} description = {Software by Frode Hegland, featuring the voice of Janine Earl 
Discontinued: htip/wwww.liquid.infolisa/)$} 

@entry | 


is also a contributor tothe first volume of ‘The Future of Text. 


Livia Polanyi} deseriptio 
Lives: NY, USA.}.$} 

Gentry | 

name = {LLM} qdescription = {*Large Language Model’ is a type of language model notable for its ability to achieve general-purpose language understanding and generation. 


LLMSs acquire these abilities by using massive amounts of data to lear billions of parameters during training and consuming large computational resources during their training and operation, 


LLMs are artificial neural networks (mainly transformers) and are (pre-)trained using self-supervised learning and semi-supervised learning. 
‘As autoregressive language models, they work by taking an input text and repeatedly predicting the next token or word. 


Up to 2020, fine tuning was the only way a model could be adapted to be able to accomplish specific tasks. Larger sized models, such as GPT-3, however, can be prompt-engineered to achieve similar results. They are thought to acquire embodied knowledge about syntax, semantics and "ontology" inherent in human language corpora, but also inaccuracies and biases present 


in the corpora 
Notable examples include OpenAl's GPT models (c.g., GPT-3.5 and GPT-4, used in ChatGPT), Google's PaLM (used in Bard), and Meta's LLaMa, as well as BLOOM, Emie 3.0 Titan, and Anthropie's Claude 2. 
hugps:/en.wikipedia.org/wiki/Large language model 29 Nov 2023}. 

@entry{ 

name = {Lue P. Beaudoin} description ~ {Faculty of Education, Simon Fraser University} §} 

@entry | 

name = {Machine leaming} description = {Machine learning (ML) is afield of inquiry devoted to understanding and building methods that ‘lear’, that is, methods that leverage data to improve performance on some set of tasks 
“Atype of Artificial Intelligence (Al). 

hutps:/en.wikipedia.org/wiki/Machine learning 

*#Technology. Timeline: 20th century.) 

@entry| 

name = {manuscript} {description 
Gentry 


{the authoring format, such as Microsoft Word, which is then either shared as-is, and stays editable, or is exported to be published in a publish format, such as PDF.}.4} 


Manuscript} $descripton = {the authoring format which is used when writing, such as Microsoft Word 
Its then either shared as-is, and stays editable, or is exported to be published in a published Document’ format, such as PDE. 

Gentry | 

name {map} (description = {View in Author. 

Here the user can place text anywhere they want and any defined concepts will automatically appear here. If there are defined concepts on the map, the user can click on them and lines will emanate to any text on the map which isin that texts definition. 4} 
@entry{ 

name = {Mark Anderson} {description = {Lndependent researcher in Hypertext and Knowledge systems. Associated with the Web & Intemet Science (WAIS) Lab at Southampton University 

Mark was heavily involved in the technical specification and ACM launch of Visual-Meta 

Mark has a background in organisational structure, knowledge, process change and data interchange 

is a co-curator of The Future of Text Symposium. 

PhD fom the University of Southampton 

Research interests includes hypertext and VR. From Southampton, UK 

hugps:/www:southampton.ac.uk/publicpolicy/support-fr-policymakers/blogs/evidence-to-policy-blog/mark-anderson-cabinet-office-placement page} 

Gentry | 

name = {Mark Bernstein} description = {Mark Bernstein is founder and Chief Scientist of Eastgate Systems. 

hutp:/www.eastgate.com/people/Bernstin html} §} 

@entry{ 

name = {Massively Multiplayer Online Role Playing Game} {description = (An interactive networked game hosting hundreds or thousands of players pursuing missions and commerce in an extended fictional universe. 4} 

@entry{ 


Matthias Maller-Prove} description = {Is a Computer scientist, interaction designer and the founder of the Chrono Research Lab, So doing lots of heritage stuff: Also sometimes refer to myselfas a kind of software archagologist. That was my introduction.” 
hugps:improve.de} 4} 

@entry| 

name = {Mesopotamia} description = 
Gentry 

name = {Meta} description = {isa large social media company which is Focusing on what they term the ‘metaverse. 
Formerly called Facebook. 

hutps:/aboutfacebook.com 

Timeline: 4 February 2004-} 4} 


{a historical region within the Tigris-Euphrates river system in the northern part of the Fertile Crescent where early writing developed (in parallel to, or influenced by/influencing Egyptian writing), including the use of colophons.|.4} 


@entry 

name = {metadata} qdescription = {information about other information, in the case of documents, this ean include structural information (headings for example), biblio} 

@entry 

name = {METADATA} description = {is information about other information, inthe case of documents, this can include structural information (headings for example), how they connect to other documents (References) and who created the document (selfciting)} 4} 
@entry| 


name = {Metaverse} description = {rom Wikipedia: “A metaverse is a network of 3D virtual worlds focused on social connection” 
“The term metaverse was coined in Neal Stephenson's 1992 science fiction novel Snow Crash” 

hutps:/en.wikipedia.org/wiki/Metaverse 

‘At the time of writing, 2022, the term has been popularised by Meta, the company previously called Facebook. Most often used in conjunction with AR/VR augmented environments, not such much the 3D worlds where the user access the world through a flat screen} 

@entry{ 

name = {Mez Breeze} description = {While studying for a degree in Applied Social Science inthe early 1990's, she started using the Intemet to create interactive digital works 

Since then, Mez has published over 300 seminal works including award-winning electronic based writing, Virtual Reality literature, XR sculptures, Artificial Intelligence artworks and books, games, and other genre-defying output all while teaching/mentoring, archiving, and supporting digital art and electronic literature. 
hutps:/mezbreeze itch i porteaits-volume-one htps:/www.mezbreezedesign.com} 4} 

@entry{ 


Michael Bonfert} description = {Researcher, Digital Media Lab, University of Bremen 
‘www.uni-bremen.de/dmlab} 


@entry{ 

name = {Michael Pidd} description = {Director ofthe Digital Humanities Institute, University of Sheffield. He isthe Pl (UK) for the C21 Editions project. 4} 
@entry | 

name = {Mini-Me} ¥description = {An “Austin Powers” movie reference.} } 


@entry 
@entry{ 
niame = {Mnemonic ID's} description = (Names like Purple Elk that are readily committed to memory without having o be written down, which usually take the form ofa short memorable gramatically valid phrase or description that can also obviate the need fora distnet password, ci 


Mixed/Extended XR} {description = {A hybrid of the VR and AR in which the system maintains a 3-D model of its surroundings permitting it to retexture or hide actual objects or simulate the insertion of virtual objects into the real environemnt such that they obey the laws of physics and are potentially indistinguishable for actual objects they might depict.) §} 


{jefRaskinTheHumanel} 4} 
@entry{ 


Names} description = {All Names in a document. 
View in Author and in Reader (with Visual-Meta) to show all named entities (and headings) 4) 

Gentry, 

name = {Navigable} qdescription = {An experience in which one can change one’s viewpoint andior orientation. 

@entry{ 

name = {Nayana Prakash} {description = {DPhil Researcher, Oxford Internet Insttute« 

Presented atthe 2023 Future of Text Sympsium.} 4} 

@entry{ 

name = {neural chip} {description = {A ditect brain implant that interfaces with brain cells by stimulating them electrically (eg. a visual cortex interface that tricks the brain into thinkng its receiving senory input from one's eyes).} 4} 
@entry| 

name = {NIC} {description = (Engelbart Concept: Networked Improvement Community “Consider an "Improvement Community 
that is consciously and effectively employing best-possible DKR (Dynamic Knowledge Repository) development and usage.” 
{augmenting society's collective 1Q).}.$} 

@entry| 

name = {NLS} description = (From Wikipedia: “NLS, or the “oN-Line System", was a revolutionary computer collaboration system developed in the 1960s. Designed by Douglas Engelbatt and implemented by researchers atthe Augmentation Research Center (ARC) atthe Stanford Research Institute (SRI). The NLS system was the first to employ the practical use of| 
hypertext links, the mouse, raster-scan Video monitors information organized by relevance, screen windowing, presentation programs, and other modern computing concepts. It was funded by ARPA (the predecessor to Defense Advanced Research Projects Agency), NASA, and the US Air Force.” https/ien.wikipedia.org/wiki/NLS_ (computer system) 

SRI sold NLS to Tymshare in 1977 and renamed it Augment.} 4} 

@entry| 

name = {noise canceling earphones} qdescription = {Earphones that generate an inverse waveform to cancel out environment sounds.) 

Gentry 

name = {NYU} fescription = {(New York University) is a private research university in New York City. Chartered in 1831 by the New York State Legislature, NYU was founded by a group of New Yorkers led by then-Secretary ofthe Treasury Albert Gallatin. 

hutps:/en.wikipedia.org/wiki/New_York University} } 

@entry{ 

name = {OpenAl} {description = {is an artificial intelligence (Al) research laboratory. 

“...consisting ofthe for-profit corporation OpenAl LP and its parent company, the non-profit OpenA Inc 

The company, considered a competitor to DeepMind, conducts research in te field of AL withthe stated goa! of promoting and developing friendly AL in a way that benefits humanity asa whole. The organization was founded in San Francisco in late 2015 by Elon Musk, Sam Altman, and others, who collectively pledged USS1 billion. Musk resigned from the board in 
February 2018 but remained a donor: In 2019, OpenAL LP received a USS! billion investment from Microsoft." 

hutps:/en.wikipedia.org/wiki/OpenAL 

higps:/opensi.com 

Products include: GPT-3 and DALL-E. 

Founded by Elon Musk, Sam Altmanllya Sutskever, Greg Brockman, Wojciech Zaremba and John Schulman, 

Timeline: 11 December 2015-.}.4} 

@entry{ 

name = {OpenAl} description = {is an artificial intelligence (AI) research laboratory consisting of the for-profit corporation OpenAI LP and its parent company, the non-profit OpenAl Inc. 

Products: GPT. GPT-2. GPT-3 

hugps:/opensi.com 

hutps:/en.wikipedia.org/wiki/Open AL} } 


(IC) as collectively engaged in improving an agreed-upon set either of individual capabilites, or of collective group capabilities-e.g. a professional society. Let's introduce a new category, a "Networked Improvement Community" (NIC): an IC 


@entry | 
name = {Orientation} description = {Roll, Pitch, and Yaw values.}.$} 

@entry 

name = {Paper} qdescription = {is a general term for a student or academic document in general 


Primarily in PDF when published or handed in 
In manuscipteditabe'personal form it is general in the Microsoft Word format.j 

Gentry, 

name = {pass-through video} description = { When wearing a headset that prevents you from seeing the room around you, pass-through video relies on cameras aligned with your left and right eyes o feed a video image of what that eye would see ifone was not wearing the display to that eye to create the illusion that you are looking at your souroundings without having 
the display on.) 1} 

@entry{ 

name = {Patrick Lichty} qdescription = {Assistant Professor, Animation/Multimedia Design, College of Arts and Creative Enterprises, Zayed University, Abu Dhabi} $} 

@entry{ 

name = {Patrick Lichty} qdescription = {Assistant Professor, Animation/Multimedia Design, College of Arts and Creative Enterprises, Zayed University, Abu Dhabi} $} 

Gentry, 

name = {PDA} {description = {*Personal Digital Assistant’. This was before smartphones.) 4)) 

@entry| 

name = {PDF} {description = {*Portable Digital Format’ developed by Adobe, now free with no license restrictions. It isa print to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for users to interact with the document in useful ways, while staying compatible with the basic PDF format. 

Its an export format rather than a manuseriptworking format.) 


@entry 

name = {PDF} {description = {*Portable Digital Format’ developed by Adobe, now free with no license restrictions. Iisa print to digial medium with few digital affoardances which my work on Visual-Meta expands to allow for users to interact with the document in useful ways, while staying compatibel with the basie PDF format.}.$} 
@entry| 

name = {perspectives} {description = {how you look} 4) 

@entry 


name {Peter J. Wasiko} {description = {Peter J. Wasilko, Esq, is an Attomey, Programmer, and Independent Scholar residing in Ossining, New York (one of the historic river towns overlooking the Hudson an hour North of New York City). 
Future Text Lab contributor. 

hitps:/futuretextlab.info/2022/12/27 /peter)-wasilko-esq/ 

Timeline: -. Lives in: New York, New York, USA.}.} 

@entry| 

name = {Phil Gooch} descriptor 
hitps:/évww:schotarey.com} 4) 


(Founder & CEO Scholarcy. 


@entry 

name = {Philipp Stephan} ‘(description = (Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science} 4) 
@entry| 

name = {photo-realistic} description = {Indistinguishable from a photograph. 4} 

@entry 


name = {Pick and Drop} qdescription ~ {A mean of moving data between devices by detecting the presence of a “Pen” gadget, assocating the network addrees ofthe data with the ID of the “Pen” and then detecting when the "Pen" is tapped on a receiving device, where its ID is looked up over the network to retrieve a copy of the original data} 4} 
@entry 

name = {Pluck and Plop} description = {A pair of gestures for transferring content across targets in VR.}.4} 

Gentry | 

name = {Principle of Severability}, description = {The notion that people won't invest in creating user generated content ifthey can’t get it out ofthe environment in which they ereate it} {cite = {10.4018/978-1-60960-567-4.ch29} 4) 

@entry 


Principle of Transferability} description ~ {The notion that fr text (and other content to realize is full untlity in VR it must be easy for users to fluidly import and export it and to transfer it between platfroms and virtual worlds. $} 
@entry{ 

name = {Purple Elk} description = {Humane / Mnemonic ID's inpsired by Purple Numbers (Doug Engelbarts fine grained hyperlinking approach) and Jeff Raskin’s work. } eit 
@entry| 

name = {QR Code} description = {A 2-D variant of a Bar Code suitable for encoding @ URL that is widely supported by cell phones, tablets, and laptops.}.4} 


{jefRaskin/TheHumanel} 4} 


Gentry| 


name = {Rachel Horst} {description = {Faculty of Education, University of British Columbia} 
@entry{ 

name = {Reader} description = fa PDE viewer from the Augmented Text company. 
hutps:/wwwaugmentedtext.infolreader} 4} 

Gentry | 


Reader} {description = {a PDF viewer from the Augmented Text company. 
hitps:/vwoaugmentedtext.info/reader} 4} 

@entry 

name {Reader Software} description = {is a minimalist PDF viewer for macOS. 
‘Can read any PDF and can provide added interactions if the PDF has Visual-Meta attached. This can either be produced by Author or any other word processor with Visual-Meta capability, or downloaded from an online repository which features Visual-Meta, such as the ACM digital library, Produced by The Augmented Text Company LTD, with programming by Jacob 
Hazelgrove. 

hitps:/Avwo:augmentedtext.info 

‘Software | Timeline: 12 July 2019-}.§} 

@entry 

name = {Real World} descriptor 
@entry 

name = {Reference Section}, description = 
@entry| 

name = {References} description = {is a lst of all the citations a document uses, in an Appendix. In-Body citation, point to these References. This language is not fixed, 


= (The world as we experience it without any computer mediation.) 4} 


{is a section to list all the sources of the citations in the article or paper} 


is sometimes used interchangably with Bibliography but in my context a Bibliography is alist of work not expressly cited but which are relevant. 
in this context “Reference” with uppercase ‘R’ refers to the appendix in an academic document which lists cites sources. In contrast, the citation inthe body of the document is referred to as in-body citation.) 4} 

@entry{ 

name {Rob Swigat} description = {American novelist, poet, short story write, futurist, and archaeology scholar best known for his satirical work, archaeology writing, science fiction, and interactive novel computer game, Portal (Activision, 1986). He is the author of sixteen books, including fourteen novels, one business book, and one translated prose poem. 
Future Text Lab contibutor. 

hutpsl/en wikipedia.org wikiRob_Swigart}§} 


@entry{ 

name = {robust} description = {is defined as the metadata being available for as long asthe document is readable, even ifthe reader software changes o the data format ofthe document changes. 
wo 

@entry{ 

name = {Rollo Carpenter} description ~ {Artificial Intelligence Developer of Cleverbot 

hutps:/en.wikipedia.org/wiki/Rollo. Carpenter} } 

Gentry, 

name = {Rome} {description = fa city in Kaly}.$} 

@entry{ 

name = {Russia} description = {or the Russian Federation, is a transcontinental country spanning Eastern Europe and Norther Asia, It isthe largest country in the world} 4} 
@entry| 


name = {Sam Brooker} description = {is an academic specialising in digital communication. 

Associate Professor of Digital Communications at Richmond American University London, UK. 

His research explores the relationship between digital technologies & theories of literature and culture. He isa regular contributor to ACM Hypertext and Social Media, frequently serving on the programme committee. Additional contributions have featured atthe ICDS and SHARP conferences. lama member ofthe PRCA International University Advisory Group and 
academic reviewer for numerous journals 

hutps:/www:tichmond.ac-uk/school-of-communications-ars-social-sciences/desam-brooker 

Timeline: -. Research interests: Digital cultures. Electronic literature. Book history. Digital humanities, Theories of authorship. Transmedia Lives in: London, UK. |} 

@entry{ 

name = [Sam Winston} ‘description = {Anist. His practice is concemmed with language not only asa cartier of messages but also as a visual form in and of itself. Initially known for his typography and artist's books he employs a variety of different approaches including drawing, performance and poetry. 
hugps:/wwwsamvinston.com) 

Gentry | 

name = {Scott Rettberg} {description = (Researcher and writer teaching digital culture and electronic literature atthe University of Bergen, Norway. 

huups:twiter com/scottetberg 


Timeline: -}.4} 

@entry{ 

name = {Search} qdescription = {Search in this context refers to instant search enabled by Liquid. } 4} 

Gentry 

name = {Shaduan Babbili} {description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science} } 


@entry{ 

name = {Sophie Whittle} description = {Research Associate forthe C21 Editions project atthe Digital Humanities Institute, University of Sheffield, She recently completed her PhD in historical linguistics and isan organiser at Sheffield Feminist Archive in her spare time. 

sophiewhittle.co.uk.}.§) 

@entry| 

name = {Spain} description = {https//en.wikipedia.org/wiki/Spain} 4} 

Gentry | 

name = {Spatial Computing} description = (“was defined in 2003 by Simon Greenwold, as “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces”. With the advent of consumer VR, AR and mixed reality, companies use ‘spatial computing’ in reference to the practice of using physical actions (head and body 
movements, gestures, speech) as inputs for interactive digital media systems, with perceived 3D physical space as the canvas for video, audio, and haptic outputs. Its also tied tothe concept of digital twins" 

The terminology used by Apple for Vision Pro. 

hutps:/en.wikipedia.org/wiki/Spatial_computing 


a 


entry 

name = {Speech to Text} description = {Analog Digital Conversion} 4} 

Gentry | 

name = {Structural} description ~ finherent characteristics such as pages, headings et.) 4} 

@entry{ 

name = {Student} description = {for my work, the student is the primary user of Author for writing and Reader for reading.} 4} 

@entry| 

name = {Sub-Spaces} description = {This can be a sub-volume within a conataining volume or in the case of a Benediktine Cyberspace, a set ofaltemate dimensions contained within a selected point in its enclosing space.}.4} 
Gentry | 

name = [Tangible Citations} description ~ (The simulation of visible links between references in the 3-D space inspired by the side-by-side text visualizations envisioned by Ted Nelson's Xanadu Project} 
@entry{ 

name = [Tangible Hypercards} description = {A passive form of Pick and Drop using cards with a QRCodes and Mnemonic ID's to eliminate the need for custom hardware.| 4} 
@entry | 

name = {Teacher} description = {in this context it the primary reader ofa student paper.) 4} 

Gentry | 

name = {Ted Nelson} faltsnamel = (Theodor Holm Nelson} description = {coined the term “hypertext}$} 

@entry{ 

name = {Text} {description = {isthe basic “stuf” of this work.}.4} 

Gentry | 

name = The Construct} description = {A self-contained small scale vitual world variously depicted as an infinite white void or martial ats Dojo in The Matrix films.) } 
@entry| 


name = {The Future of Text} description 
hitps:/futuretextpublishing.comy 4} 
@entry{ 

name = {The Future of Text} description 
hitps:/futuretextpublishing.comy. 4} 
@entry | 

name = {The Matix} descriptios 
@entry{ 

name = {Theresa Berg} description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science.) 
@entry| 

name = {Time Index}. description = {A reference to a point in time or rage of times within a time-varying dataset.) 9} 

Gentry | 

name = {Timo Littig} description = {Student of Prof: Dr. Alexander Mehler and Giuseppe Abrami at Goethe University Frankfurt, Computer Science} §} 
@entry{ 

name = {tool for thought} description = {is both a book and a category of tools, primarily software tools, to augment how people think. 


{Annual Symposium and Book Series (first volume published 2020) as well as community for fostering dialogue around the future of text which I started over a decade ago and which is often co-hosted or presented by Vint Cert. 


{Annual Symposium and Book Series (first volume published 2020) as well as community for fostering dialogue around the future of text which I started over a decade ago and which is often co-hosted or presented by Vint Cert. 


= {A series of science fiction movies in which people are grown in pods with direct Neural Chip interfaces to a virtual world depicting America in the 1990's that is also populated by Actificially Intelligent “Agents”.} 4} 


The book: “Tools for Thought: The History and Future of Mind-Expanding Technology is a work of "retrospective futurism’ in which Smart Mobs author Howard Rheingold looked atthe history of computing and then attempted to predict what the networked world might look like in the mid-1990s. The book covers the groundbreaking work of thinkers like Alan Turing, 
John von Neumann, and .C.R. Licklider, as well as Xerox PARC, Apple Computer, and Microsoft (when Microsoft was "aiming forthe hundred-million-dllar category"). Rheingold wrote thatthe impetus behind Tools for Thought was to understand where "mind-amplifying technology" was going by understanding where it came from.” htps(/en.wikipedia.org/wiki/ 
Tools_for_ Thought} 

@entry{ 

name = [Touch}.qdescription = (Touching a device to interact} 

@enry 

name = {UK} {description = {hups://en.wikipedia.org/wiki/United_Kingdom}.$} 

Gentry, 

name = {Ukraine} description = {hups/ien.wikipedia.org/wiki/Ukraine 

*#Location} 4} 

@entry{ 

name = {University of Southampton} qdeseription = {University inthe UK.) 4} 

@ontry 

name = {USA f¥description = fhups:!ien wikipedia org/wiki/United States} 

Gentry 

name = { Viewpoint} description = {X, Y, and Z coordinates in a 3-D space.}.$} 

@entry{ 

name = { Viewspec} description = {A data structure containing all of a visualization’s control parameters including referenced data sets needed to reproduce that visualization from a given perspective.) 
@entry{ 

name = { ViewSpec} description = { Viewspees ~ short for "view specifications”. Allows you to control your view of a document. 

hugps:/www.dougengelbart.org/contentview/218)},4} 


Gentry 

name = { ViewSpec} “description = {Engelbart Notion: View Specifications fora user to see their workin different views to gain new perspectives and to navigate their document.}.§} 
@entry| 

name = { Viewspee Reification} description ~ {Encapsulating a viewspec as a manipulable object.) 4} 

@entry| 

name = { ViewSpecs} {description ~ { Viewspees ~ short for "view specifications" ~ allow you to control your view of a file, 


hutps:/‘vwwdougengelbartorg/contentview/218/}.} 

@entry 

name = {Vint Cerf} falt-namel = {Vinton Gray Cerf} description ~ {Co-inventor of the Internet, VP and Chief Internet Evangelist for Google. Chairman of the Marconi Society. Former executive at MCI, the Corporation for National Research Initiatives, the Internet Corporation for Assigned Names and Numbers (ICANN), the American Registry for Intemet Numbers, 
(ARIN), the Association for Computing Machinery (ACM) and member of the Faculty of Stanford University. Fellow of IEEE, ACM, BCS, AAAS, American Academy of Arts and Sciences, and American Philosophical Society. Member of the US National Academies of Engineering and Science and foreign member of the Royal Society and the Royal Swedish Engineering 
Society 

He has been a strong and vocal supporter of Frode Hegland’s work, and Visual-Mta in particular. 

Category: Person 

Timeline: 23 June 1943}.4} 


@entry| 

name = {Vision Pro} fMdescription = {XR headset. Made by Apple. Refered to as Spatial Computing. Runs visionOS.} 4} 
@entry 

name = {visionOS} {description = {Apple’s XR operating system, frst deployed on the Vision Pro.| 4) 


Gentry 

name = { Visual-Meta} {description ~ fis my approach for embedding metadata in documents, initially PDE, to allow viewer software to provide tich interactions. The Visual-Meta is added as an appendix to the end ofthe document in a format inspired by BibTex. 
butp:/visual-meta. info} §} 

@entry| 

name = { Visual-Meta} {description = (An open and robust way to augment flat PDF documents to make them more interactive, by The Augmented Text Company people 

hutp/visual-meta. info} §} 

@entry{ 

name = { Visual-Meta} description = {A open and robust way to augment flat PDE documents to make them more interactive, by The Augmented Text Company people 

butp/visual-meta. info} §} 

Gentry | 

Gentry | 


Voice} {description = {Speaking and listening for interaction.) 4} 


name {VR} idescription ~ {*Virtual Reality’ isa simulated experience that employs pose tracking and 3D near-eye displays to give the user an immersive feel ofa virtual world (Wikipedia) 
In contrast to AR where the user can see their physical environment. 

Included in the term XR.}.4}, 

@entry| 

name = {VR} {description = {*Virtual Reality’ isa simulated experience that employs pose tracking and 3D near-eye displays to give the user an immersive feel ofa virtual world (Wikipedia) 
Included in the term XR.}.9}, 


@entry{ 

name = {Wendy Hall} description ~ {Frode Hegland’s PhD primary supervisor, along with Les Carr and David Millard, §} 

Gentry | 

name = {Xbox} {description = {A popular high end video game system.) 

@entry| 

name = {XR} description ~ {‘eXtended Reality’, aterm including both Virtual Realty (VR) and Augmented Realty (AR), This isa useful term since headsets increasingly Feature passthrough video, allowing the user to toggle between VR and AR modes.}.§} 
@entry{ 

name = [XR Fragments} {description = {A mechanism for linking VR Experiences on the web so clicking on a virtual object can effect a perspective or scene change analogous to clicking @ hypertext link to follow it on an ordinary web page.}.4} 
Gentry | 

name = { Yannick Heinrich} {description = {Student of Prof. Dr. Alexander Mehler and Giuseppe Abram at Goethe University Frankfurt, Computer Science.},4) 

Gentry | 


name = {Yohanna Joseph Waliya} fdescription = {Enseignant et chercheur & l'université de Calabar. 
UNESCO Janusz Korezak Fellow, ELO Research Fellow, Winner of Janusz Korezak Prize for Global South 2020, Creator & Curator of MAELD & ADELD. Executive Director of AELA & ADELL Auteur de La récolte de vie, Climatophosis (Winner Best Use of DH for Fun 2020) ete. 
hutps:/atricanelit.org 

htps:/oreid.org/0000-0003-4456-7458 

https:/anical-ng.academia.edu/WALIYAYOHANNAJOSEPH, 

hutps://www:tesearchgate.net/profile/Yohanna_Waliya} 


{glossary-end} 


@fendnotes-start} 


@entry(1, 
text= {Generated with the prompt “futuristic gold metal robot with many cables and swirls of light, reading a book witha grey cover against a white background, called "The future of text. Many other books floating nearby” with www.imagine.art using Imagine V4} 
Gentry 2, 

text = {https:/webb :nasa.gov/contentscienceifirstLight. html} §} 

Gentry 3, 

text= {Is the universe twice as old as we thought? 

hutps:/cosmosmagazine.com/space/astrophysies/universe.27-billion-years-old/} 4} 

@ontry'4, 

text = {hutps:/www.livescience.com/spacelastronomy/james-webb-telescope-finds-an-extreme-glow-coming-fom-90-of-he-universes-earliest-galaxies} 

@ontry{5, 

text= {As first proposed by Doug Engelbart.} 4) 

entry (6, 

text = {hutps://youtu.belGEPhLqwK o6g?si-tMm ¥jOLLig31b35}.$} 

@entry (7, 

text = {https:!/www.dropbox.comvs/pnalfvfSsebetwg xrOS%43A%20What®42010%20Expect®420F rom®%420the®20Software’é20Designed”420for%420Apple%427s%420A R%3AVRY%420Headset%420-9420MacRumors pda} 

Gentry 8, 


text = {What performance-enhancing stimulants mean for economic growth 
hugps:/Avww.economist.com/ 
hugps:/4vww.economistcomyinance-and-economics/2023/05/25/what-performance-enhancing-stimulants-mean-for-economic-growth} 4} 
@entry (9, 

text= (Prepare forthe Textpocalypse By Matthew Kirschenbaum in The Atlantic. 

hugps:/vww.theatlantie, com technology/archive/2023/03/ai-chatgptwrting-language-models/673318/2utm_source-apple_news) 4} 
@enry (10, 


text= thtps:/speechify.com) 4} 
@entry 11, 
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